Hinge Loss Function Research Articles

Large-scale pinball twin support vector machines

Twin support vector machines (TWSVMs) have been shown to be effective classifiers for a range of pattern classification tasks. However, the TWSVM formulation suffers from a range of shortcomings: (i) TWSVM uses hinge loss function which renders it sensitive to dataset outliers (noise sensitivity). (ii) It requires a matrix inversion calculation in the Wolfe-dual formulation which is intractable for datasets with large numbers of features/samples. (iii) TWSVM minimizes the empirical risk instead of the structural risk in its formulation with the consequent risk of overfitting. This paper proposes a novel large scale pinball twin support vector machines (LPTWSVM) to address these shortcomings. The proposed LPTWSVM model firstly utilizes the pinball loss function to achieve a high level of noise insensitivity, especially in relation to data with substantial feature noise. Secondly, and most significantly, the proposed LPTWSVM formulation eliminates the need to calculate inverse matrices in the dual problem (which apart from being very computationally demanding may not be possible due to matrix singularity). Further, LPTWSVM does not employ kernel-generated surfaces for the non-linear case, instead using the kernel trick directly; this ensures that the proposed LPTWSVM is a fully modular kernel approach in contrast to the original TWSVM. Lastly, structural risk is explicitly minimized in LPTWSVM with consequent improvement in classification accuracy (we explicitly analyze the properties of classification accuracy and noise insensitivity of the proposed LPTWSVM). Experiments on benchmark datasets show that the proposed LPTWSVM model may be effectively deployed on large datasets and that it exhibits similar or better performance on most datasets in comparison to relevant baseline methods.

Twin [formula omitted]-class support vector classification with pinball loss

The twin K-class support vector classification (TKSVC) is an effective and efficient algorithm for multi-class classification problem. However, due to the use of hinge loss function, it is very sensitive to feature noises and not stable for re-sampling. To avoid the above flaws, we present a novel twin K-class support vector classification with pinball loss (Pin-TKSVC) in this paper. The structure of Pin-TKSVC is analogous to that of TKSVC. Namely, it adopts “One-vs.-One-vs.-Rest” structure to do multi-class classification and solves two smaller sized quadratic programming problems to save the running time. Besides, the introduction of pinball loss does not increase the additional computational complexity. More importantly, the Pin-TKSVC maximizes the quantile distance between different categories, making it a more robust classifier. To testify its performance, we do numerical experiments on twenty datasets with different noises, and compare it with five state-of-the-art algorithms. The experimental results confirm the validity of our algorithm.

Stochastic Subgradient for Large-Scale Support Vector Machine Using the Generalized Pinball Loss Function

In this paper, we propose a stochastic gradient descent algorithm, called stochastic gradient descent method-based generalized pinball support vector machine (SG-GPSVM), to solve data classification problems. This approach was developed by replacing the hinge loss function in the conventional support vector machine (SVM) with a generalized pinball loss function. We show that SG-GPSVM is convergent and that it approximates the conventional generalized pinball support vector machine (GPSVM). Further, the symmetric kernel method was adopted to evaluate the performance of SG-GPSVM as a nonlinear classifier. Our suggested algorithm surpasses existing methods in terms of noise insensitivity, resampling stability, and accuracy for large-scale data scenarios, according to the experimental results.

The Novel Sequence Distance Measuring Algorithm Based on Optimal Transport and Cross-Attention Mechanism

In this paper, we propose a novel sequence distance measuring algorithm based on optimal transport (OT) and cross-attention mechanism. Given a source sequence and a target sequence, we first calculate the ground distance between each pair of source and target terms of the two sequences. The ground distance is calculated over the subsequences around the two terms. We firstly pay attention from each the source terms to each target terms with attention weights, so that we have a representative source subsequence vector regarding each term in the target subsequence. Then, we pay attention from each representative vector of the term of the target subsequence to the entire source subsequence. In this way, we construct the cross-attention weights and use them to calculate the pairwise ground distances. With the ground distances, we derive the OT distance between the two sequences and train the attention parameters and ground distance metric parameters together. The training process is conducted with training triplets of sequences, where each triplet is composed of an anchor sequence, a must-link sequence, and a cannot-link sequence. The corresponding hinge loss function of each triplet is minimized, and we develop an iterative algorithm to solve the optimal transport problem and the attention/ground distance metric parameters in an alternate way. The experiments over sequence similarity search benchmark datasets, including text, video, and rice smut protein sequence data, are conducted. The experimental results show the algorithm is effective.

Multi-label classification with Missing Labels using Label Correlation and Robust Structural Learning

A class of machine learning problem where each instance may either belong to one or more than one class simultaneously is known as Multi-label classification problem. Unlike other classification problems there exist several challenges and the important among them is learning the label correlation when labels are missing. In this paper, we present a unified learning system that addresses the aforementioned issue, and suggest a novel multi-label classifier termed as Multi-label classification with Missing Labels using Label Correlation and Robust Structural Learning (MLLCRS-ML). The propose classifier considers label-specific features along with utilizing the structural property of data and pairwise label correlation (both positive and negative label correlations) for recovering the missing labels. We have use hinge loss function which ensures less sensitivity towards outliers and the accelerated proximal gradient method (APG) to efficiently solve the underlying optimization problem. Experimental results on several benchmark data sets show that our propose approach MLLCRS-ML is as competitive as other state-of-the-art approaches.

Indefinite twin support vector machine with DC functions programming

Twin support vector machine (TWSVM) is an efficient algorithm for binary classification. However, the lack of the structural risk minimization principle restrains the generalization of TWSVM and the guarantee of convex optimization constraints TWSVM to only use positive semi-definite kernels (PSD). In this paper, we propose a novel TWSVM for indefinite kernel called indefinite twin support vector machine with difference of convex functions programming (ITWSVM-DC). The indefinite TWSVM (ITWSVM) leverages a maximum margin regularization term to improve the generalization of TWSVM and a smooth quadratic hinge loss function to make the model continuously differentiable. The representer theorem is applied to the ITWSVM and the convexity of the ITWSVM is analyzed. In order to address the non-convex optimization problem when the kernel is indefinite, a difference of convex functions (DC) is used to decompose the non-convex objective function into the subtraction of two convex functions and a line search method is applied in the DC algorithm to accelerate the convergence rate. A theoretical analysis illustrates that ITWSVM-DC can converge to a local optimum and extensive experiments on indefinite and positive semi-definite kernels show the superiority of ITWSVM-DC.

On Lagrangian L2-norm pinball twin bounded support vector machine via unconstrained convex minimization

With the introduction of the regularization term in the formulation of the well-known twin support vector machine (TWSVM) for classification , twin bounded support vector machine (TBSVM) method was proposed recently as an improved version by implementing the structural risk minimization principle. However, TBSVM employs hinge loss function and it is sensitive to noise and unstable to re-sampling. Since the pinball loss function related to quantile distance enjoys noise insensitivity property , a novel TBSVM method with squared pinball loss function for classification is proposed. The noise insensitivity and scatter minimization properties are discussed. Our formulation is further simplified as a pair of unconstrained strongly convex minimization problems in the dual space free of matrix inversion terms and having only m variables where m is the number of training examples. As opposed to TWSVM and TBSVM wherein approximate kernel generated surfaces are constructed, kernel trick is applied directly in our formulation and thereby elegant formulation as in the classical support vector machine (SVM) is achieved. Numerical experiments performed on a synthetic and thirteen benchmark datasets with noise where better or comparable generalization performance with faster learning speed by the proposed method confirms its suitability and applicability to problems of interest.

Multi-atlas classification of autism spectrum disorder with hinge loss trained deep architectures: ABIDE I results

The Autism Brain Imaging Data Exchange (ABIDE) database has emerged as a benchmark cohort for training and testing machine learning models for classifying autism. Considering recent successes using deep learning, increasing the overall classification accuracy on this dataset is a challenging endeavor. In this paper, we propose a multi-input deep neural network model for classifying autism. Our model’s architecture is designed to incorporate neuroimaging data preprocessed using three different reference atlases. For each training example, the proposed deep neural network simultaneously receives connectome data originating from three different parcellation strategies and automatically learns discriminative features from the three input sets. This process results in learned features that are more general and less dependent on a single brain parcellation method. Moreover, the proposed neural network is trained using the hinge loss function, which strongly penalizes misclassifications. We validate our model under cross-validation frameworks, using a set of 1,038 real participants and an augmented set of 10,038 samples. We achieve a classification accuracy of 78.07% on real data and 79.13% on augmented data, which is about 9% higher than previously reported results. Our results reinforce the idea that discordance in resting-state network connectivity constitutes a major discriminatory feature between patients with autism and healthy individuals. Our proposed multi-atlas deep neural network classification framework is potentially applicable to additional brain disorders.

Long-term Cognitive Network-based architecture for multi-label classification

This paper presents a neural system to deal with multi-label classification problems that might involve sparse features. The architecture of this model involves three sequential blocks with well-defined functions. The first block consists of a multilayered feed-forward structure that extracts hidden features, thus reducing the problem dimensionality. This block is useful when dealing with sparse problems. The second block consists of a Long-term Cognitive Network-based model that operates on features extracted by the first block. The activation rule of this recurrent neural network is modified to prevent the vanishing of the input signal during the recurrent inference process. The modified activation rule combines the neurons’ state in the previous abstract layer (iteration) with the initial state. Moreover, we add a bias component to shift the transfer functions as needed to obtain good approximations. Finally, the third block consists of an output layer that adapts the second block’s outputs to the label space. We propose a backpropagation learning algorithm that uses a squared hinge loss function to maximize the margins between labels to train this network. The results show that our model outperforms the state-of-the-art algorithms in most datasets.

Sparse Twin Support Vector Clustering Using Pinball Loss.

Clustering is a widely used machine learning technique for unlabelled data. One of the recently proposed techniques is the twin support vector clustering (TWSVC) algorithm. The idea of TWSVC is to generate hyperplanes for each cluster. TWSVC utilizes the hinge loss function to penalize the misclassification. However, the hinge loss relies on shortest distance between different clusters, and is unstable for noise-corrupted datasets, and for re-sampling. In this paper, we propose a novel Sparse Pinball loss Twin Support Vector Clustering (SPTSVC). The proposed SPTSVC involves the ϵ-insensitive pinball loss function to formulate a sparse solution. Pinball loss function provides noise-insensitivity and re-sampling stability. The ϵ-insensitive zone provides sparsity to the model and improves testing time. Numerical experiments on synthetic as well as real world benchmark datasets are performed to show the efficacy of the proposed model. An analysis on the sparsity of various clustering algorithms is presented in this work. In order to show the feasibility and applicability of the proposed SPTSVC on biomedical data, experiments have been performed on epilepsy and breast cancer datasets.

Time-weighted Fuzzy Support Vector Machines for classification in changing environments

The predictive performance of classification methods relies heavily on the nature of the environment, as in the joint distribution of inputs and outputs may evolve over time. This issue is known as dataset shift. Given that most statistical and machine learning techniques assume that the training sample is drawn from the same distribution as the test data used for evaluation, an appreciable amount of researchers and practitioners tend to ignore this issue at the model construction stage. In this paper, we propose a novel Fuzzy Support Vector Machine strategy, in which the traditional hinge loss function is redefined to account for dataset shift. Additionally, we propose a general version of this loss function applying aggregation operators in order to improve performance by dealing with dataset shift via fuzzy logic. Originally developed as linear approaches, our proposals are extended to kernel-based classification for non-linear machine learning. Our methods are able to perform best compared to traditional classifiers in terms of out-of-time prediction using simulated and real-world dataset for credit scoring, confirming the theoretical virtues of our approach.

L1-norm Laplacian support vector machine for data reduction in semi-supervised learning

As a semi-supervised learning method, Laplacian support vector machine (LapSVM) is popular. Unfortunately, the model generated by LapSVM has a poor sparsity. A sparse decision model has always been fascinating because it could implement data reduction and improve performance. To generate a sparse model of LapSVM, we propose an $$\ell _1$$ -norm Laplacian support vector machine ( $$\ell _1$$ -norm LapSVM), which replaces the $$\ell _2$$ -norm with the $$\ell _1$$ -norm in LapSVM. The $$\ell _1$$ -norm LapSVM has two techniques that can induce sparsity: the $$\ell _1$$ -norm regularization and the hinge loss function. We discuss two situations for the $$\ell _1$$ -norm LapSVM, linear and nonlinear ones. In the linear $$\ell _1$$ -norm LapSVM, the sparse decision model implies that features with nonzero coefficients are contributive. In other words, the linear $$\ell _1$$ -norm LapSVM can perform feature selection to achieve the goal of data reduction. Moreover, the nonlinear (kernel) $$\ell _1$$ -norm LapSVM can also implement data reduction in terms of sample selection. In addition, the optimization problem of the $$\ell _1$$ -norm LapSVM is a convex quadratic programming one. That is, the $$\ell _1$$ -norm LapSVM has a unique and global solution. Experimental results on semi-supervised classification tasks have shown a comparable performance of our $$\ell _1$$ -norm LapSVM.

Smooth twin bounded support vector machine with pinball loss

The twin support vector machine improves the classification performance of the support vector machine by solving two small quadratic programming problems. However, this method has the following defects: (1) For the twin support vector machine and some of its variants, the constructed models use a hinge loss function, which is sensitive to noise and unstable in resampling. (2) The models need to be converted from the original space to the dual space, and their time complexity is high. To further enhance the performance of the twin support vector machine, the pinball loss function is introduced into the twin bounded support vector machine, and the problem of the pinball loss function not being differentiable at zero is solved by constructing a smooth approximation function. Based on this, a smooth twin bounded support vector machine model with pinball loss is obtained. The model is solved iteratively in the original space using the Newton-Armijo method. A smooth twin bounded support vector machine algorithm with pinball loss is proposed, and theoretically the convergence of the iterative algorithm is proven. In the experiments, the proposed algorithm is validated on the UCI datasets and the artificial datasets. Furthermore, the performance of the presented algorithm is compared with those of other representative algorithms, thereby demonstrating the effectiveness of the proposed algorithm.

Non-smooth classification model based on new smoothing technique

This work describes a framework for solving support vector machine with kernel (SVMK). Recently, it has been proved that the use of non-smooth loss function for supervised learning problem gives more efficient results [1]. This gives the idea of solving the SVMK problem based on hinge loss function. However, the hinge loss function is non-differentiable (we can’t use the standard optimization methods to minimize the empirical risk). To overcome this difficulty, a special smoothing technique for the hinge loss is proposed. Thus, the obtained smooth problem combined with Tikhonov regularization is solved using a stochastic gradient descent method. Finally, some numerical experiments on academic and real-life datasets are presented to show the efficiency of the proposed approach.

Ship Matching Using Convolutional Neural Network in Multi-source Synthetic Aperture Radar Images

Niu, L. and Lang, H., 2020. Ship matching using convolutional neural network in multi-source synthetic aperture radar images. In: Jung, H.-S.; Lee, S.; Ryu, J.-H., and Cui, T. (eds.), Advances in Geospatial Research of Coastal Environments. Journal of Coastal Research, Special Issue No. 102, pp. 166-175. Coconut Creek (Florida), ISSN 0749-0208.There are two major challenges in the ship matching and tracking task by synthetic aperture radar (SAR) images. First, images from different satellites have different incidence angles, polarization modes and resolutions, leading to the nonlinear geometric deformation of ships and the difference in image quality. The above differences result in a significant drop in the similarity of matching ships. Second, the principal direction of ships is arbitrary, which requires that the matching method has rotation-invariance. For the first challenge, the siamese-type convolution neural networks are adopted to reduce the distance between matching ships and increase the distance between non-matching ships. For the second challenge, a minimum bounding square (MBS) segmentation method is used to process all the ship images to the horizontal direction. After image preprocessing using MBS, there are only two situations for the principal direction of ships and rotation-invariance is transformed into the invariance of prow-to-stern interchange. To solve the problem of prow-to-stern interchange, a constrained hinge loss is proposed to ensure that the minimum similarity of matching images is still greater than the maximum similarity of non-matching images. There are a variety of siamese-type convolutional neural networks with good performance currently, the focus of the research is to use the two proposed strategies (MBS and constrained hinge loss) to improve the performance of existing methods instead of designing network architectures. In order to verify the feasibility of the proposed strategies, a multi-source ship target database covers different SAR sensors is constructed for ship matching. The experimental results prove that the proposed methods can match the ship target precisely and outperform the methods without MBS preprocessing and the constrained hinge loss function.

Improved Sparsity of Support Vector Machine with Robustness Towards Label Noise Based on Rescaled $$\alpha $$-Hinge Loss with Non-smooth Regularizer

As support vector machines (SVM) are used extensively in machine learning applications, it becomes essential to obtain a sparse model that is also robust to noise in the data set. Although many researchers have presented different approaches to get a robust SVM, the work on robust SVM based on rescaled hinge loss function (RSVM-RHHQ) has attracted a great deal of attention. The method of using correntropy with hinge loss function has added a noticeable amount of robustness to the model. However, the sparsity of the model can be further improved. In this work, we focus on enhancing the sparsity of RSVM-RHHQ. As this work is the improved version of the RSVM-RHHQ, we follow the same track (of adding noise in the data) of RSVM-RHHQ with altogether a new problem formulation. We apply correntropy to the $$\alpha $$ -hinge loss function, which results in a better loss function than the rescaled hinge loss function. We use a non-smooth regularizer with a non-convex and non-smooth loss function. We solve this non-smooth and non-convex problem using the primal–dual proximal method. We find that this combination not only adds sparsity to the model, but it is also better than the existing robust SVM methods in terms of robustness towards label noise. We also provide the convergence proof of the proposed approach. In addition, the time complexity of the optimization technique is included. We perform experiments over various publicly available real-world data sets to compare the proposed method with the existing robust SVM methods. For experimentation purposes, we use small data sets, large data sets, and also data sets with significant class imbalance. Experimental results show that the proposed approach outperforms existing methods in sparseness, accuracy, and robustness. We also provide the sensitivity analysis of the regularization parameter for the label noise in the data set.

Smooth Twin Bounded Support Vector Machine with Pinball Loss

The twin support vector machine improves the classification performance of the support vector machine by solving two smaller quadratic programming problems. However, for the twin support vector machine and some of its variants, the constructed models are usually transformed from the original space into the dual space to obtain the solutions. Meanwhile, the hinge loss function used in above models is sensitive to noise and unstable in resampling. In order to further improve the performance of the twin support vector machine, the pinball loss function is introduced into the twin bounded support vector machine directly, and the non-differentiable problem of the pinball loss function at zero is solved by constructing a smooth approximation function. Based on this, a smooth twin bounded support vector machine model with pinball loss is obtained. The model is solved iteratively in the original space by using the Newton-Armijo method, then a smooth twin bounded support vector machine with pinball loss algorithm is proposed. In the experiments, the proposed twin support vector machine is validated on the UCI datasets, which shows the effectiveness of the proposed algorithm.

Deep Learning Structure for Cross-Domain Sentiment Classification Based on Improved Cross Entropy and Weight

Within the sentiment classification field, the convolutional neural network (CNN) and long short-term memory (LSTM) are praised for their classification and prediction performance, but their accuracy, loss rate, and time are not ideal. To this purpose, a deep learning structure combining the improved cross entropy and weight for word is proposed for solving cross-domain sentiment classification, which focuses on achieving better text sentiment classification by optimizing and improving recurrent neural network (RNN) and CNN. Firstly, we use the idea of hinge loss function (hinge loss) and the triplet loss function (triplet loss) to improve the cross entropy loss. The improved cross entropy loss function is combined with the CNN model and LSTM network which are tested in the two classification problems. Then, the LSTM binary-optimize (LSTM-BO) model and CNN binary-optimize (CNN-BO) model are proposed, which are more effective in fitting the predicted errors and preventing overfitting. Finally, considering the characteristics of the processing text of the recurrent neural network, the influence of input words for the final classification is analysed, which can obtain the importance of each word to the classification results. The experiment results show that within the same time, the proposed weight-recurrent neural network (W-RNN) model gives higher weight to words with stronger emotional tendency to reduce the loss of emotional information, which improves the accuracy of classification.

A distributionally robust area under curve maximization model

Area under ROC curve (AUC) is a performance measure for classification models. We propose new distributionally robust AUC models (DR-AUC) that rely on the Kantorovich metric and approximate AUC with the hinge loss function, and derive convex reformulations using duality. The DR-AUC models outperform deterministic AUC and support vector machine models and have superior worst-case out-of-sample performance, thereby showing their robustness. The results are encouraging since the numerical experiments are conducted with small-size training sets conducive to low out-of-sample performance.

Ramp loss for twin multi-class support vector classification

ABSTRACTTwin K-class support vector classification (TKSVC) adopts ‘One-vs.-One-vs.-Rest’ structure to utilise all the samples to increase the prediction accuracy. However, TKSVC is sensitive to noises or outliers due to the use of the Hinge loss function. To reduce the negative influence of outliers, in this paper, we propose a more robust algorithm termed as Ramp loss for twin K-class support vector classification (Ramp-TKSVC) where we use the Ramp loss function to substitute the Hinge loss function in TKSVC. Because the Ramp-TKSVC is a non-differentiable non-convex optimisation problem, we adopt Concave–Convex Procedure (CCCP) to solve it. To overcome the drawbacks of conventional multi-classification methodologies, the TKSVC is utilised as a core of our Ramp-TKSVC. In the Ramp-TKSVC, the outliers are prevented from becoming support vectors, thus they are not involved in the construction of hyperplanes, making the Ramp-TKSVC more robust. Besides, the Ramp-TKSVC is sparser than the TKSVC. To verify the validity of our Ramp-TKSVC, we conduct experiments on 12 benchmark datasets in both linear and nonlinear cases. The experimental results indicate that our algorithm outperforms the other five compared algorithms.