Multisurface Proximal Support Vector Machine Research Articles

Towards improving the performance of blind image steganalyzer using third-order SPAM features and ensemble classifier

The success rate for blind or universal steganalysis lies in the ability to extract the statistical footprints of image features. Further, the choice of machine learning (ML) algorithm is crucial to distinguish the stego image more precisely from the untouched clean images. Literature suggests that most steganalysis approaches report less favorable detection accuracy despite considering many features. This study presents a three-step process to accurately identify the clean and stego images to solve this issue. We used the curvelet denoising as an initial phase during the first step to suppress the natural noise residuals (NRs) by producing the stego NRs. Secondly, it extracts the Third-order Markov-chain sample transition probability matrices as features. Finally, the oblique decision tree ensemble using a multisurface proximal support vector machine (SVM) classifier has been utilized to achieve greater detection accuracy than the state-of-the-art classifiers. The experiments are performed on an extensive database comprising clean and stego images generated from nine embedding schemes with varying payloads. The experimental results suggest that an accuracy of 93.12 has been achieved using the proposed Third order subtractive pixel adjacency matrix (SPAM) features with an ensemble classifier.

Oblique and rotation double random forest

Random Forest is an ensemble of decision trees based on the bagging and random subspace concepts. As suggested by Breiman, the strength of unstable learners and the diversity among them are the ensemble models’ core strength. In this paper, we propose two approaches known as oblique and rotation double random forests. In the first approach, we propose rotation based double random forest. In rotation based double random forests, transformation or rotation of the feature space is generated at each node. At each node different random feature subspace is chosen for evaluation, hence the transformation at each node is different. Different transformations result in better diversity among the base learners and hence, better generalization performance. With the double random forest as base learner, the data at each node is transformed via two different transformations namely, principal component analysis and linear discriminant analysis. In the second approach, we propose oblique double random forest. Decision trees in random forest and double random forest are univariate, and this results in the generation of axis parallel split which fails to capture the geometric structure of the data. Also, the standard random forest may not grow sufficiently large decision trees resulting in suboptimal performance. To capture the geometric properties and to grow the decision trees of sufficient depth, we propose oblique double random forest. The oblique double random forest models are multivariate decision trees. At each non-leaf node, multisurface proximal support vector machine generates the optimal plane for better generalization performance. Also, different regularization techniques (Tikhonov regularization, axis-parallel split regularization, Null space regularization) are employed for tackling the small sample size problems in the decision trees of oblique double random forest. The proposed ensembles of decision trees produce trees with bigger size compared to the standard ensembles of decision trees as bagging is used at each non-leaf node which results in improved performance. The evaluation of the baseline models and the proposed oblique and rotation double random forest models is performed on benchmark 121 UCI datasets and real-world fisheries datasets. Both statistical analysis and the experimental results demonstrate the efficacy of the proposed oblique and rotation double random forest models compared to the baseline models on the benchmark datasets.

Oblique Decision Tree Ensemble via Twin Bounded SVM

Ensemble methods with “perturb and combine” strategy have shown improved performance in the classification problems. Recently, random forest algorithm was ranked one among 179 classifiers evaluated on 121 UCI datasets. Motivated by this, we propose a new approach for the generation of oblique decision trees. At each non-leaf node, the training data samples are grouped in two categories based on the Bhattachrayya distance with randomly selected feature subset. Then, twin bounded support vector machine (TBSVM) is used to get two clustering hyperplanes such that each hyperplane is closer to data points of one group and as far as possible from the data points of other group. Based on these hyperplanes, each non-leaf node is splitted to generate the decision tree. In this paper, we used different base models like random forest (RaF), rotation forest (RoF), random sub rotation forest (RRoF) to generate the different oblique decision tree forests named as TBRaF, TBRoF and TBRRoF, respectively. In earlier oblique decision trees, like multisurface proximal support vector machine (MPSVM) based oblique decision trees, matrices are semi-positive definite and hence different regularization methods are required. However, no explicit regularization techniques need to be applied to the primal problems as the matrices in the proposed TBRaF, TBRoF and TBRRoF are positive definite. We evaluated the performance of the proposed models (TBRaF, TBRoF and TBRRoF) on 49 datasets taken from the UCI repository and on some real-world biological datasets (not in UCI). The experimental results and statistical tests conducted show that TBRaF and TBRRoF outperform other baseline methods.

Improving image steganalyser performance through curvelet transform denoising

The major challenge of feature based blind steganalysers lies in designing effective image features which give true evidence of the stego noise rather than the natural noise present in the images. Hence they report low detection accuracy in real time implementation in spite of employing 100s of features in the process. In this paper, we coin a new paradigm for detecting steganography by examining the task as a three-steps process with the following repercussions: (a) employing curvelet transform denoising as a pre-processing step that produces better stego noise residuals suppressing the natural noise residual rather than a general denoising step before feature extraction, (b) extracting various steganalytic features, both in spatial domain as well transform domain and (c) implementing the system based on an efficient classifier, multi-surface proximal support vector machine ensemble oblique random rotation forest, that provides detection rate superior to other existing classifiers. Extensive experimentation with huge database of clean and steganogram images produced from seven steganographic schemes with varying embedding rates, and using five steganalysers, shows that the proposed paradigm improves the detection accuracy substantially and proves to be a high performance strategy even at low embedding rates. This model can be employed as a preprocessing component for any image steganalyser and high performance accuracy can be obtained.

Oblique Decision Tree Ensemble via Multisurface Proximal Support Vector Machine.

A new approach to generate oblique decision tree ensemble is proposed wherein each decision hyperplane in the internal node of tree classifier is not always orthogonal to a feature axis. All training samples in each internal node are grouped into two hyper-classes according to their geometric properties based on a randomly selected feature subset. Then multisurface proximal support vector machine is employed to obtain two clustering hyperplanes where each hyperplane is generated such that it is closest to one group of the data and as far as possible from the other group. Then, one of the bisectors of these two hyperplanes is regarded as the test hyperplane for this internal node. Several regularization methods have been applied to handle the small sample size problem as the tree grows. The effectiveness of the proposed method is demonstrated by 44 real-world benchmark classification data sets from various research fields. These classification results show the advantage of the proposed approach in both computation time and classification accuracy.

Non-parallel support vector classifiers with different loss functions

This paper introduces a general framework of non-parallel support vector machines, which involves a regularization term, a scatter loss and a misclassification loss. When dealing with binary problems, the framework with proper losses covers some existing non-parallel classifiers, such as multisurface proximal support vector machine via generalized eigenvalues, twin support vector machines, and its least squares version. The possibility of incorporating different existing scatter and misclassification loss functions into the general framework is discussed. Moreover, in contrast with the mentioned methods, which applies kernel-generated surface, we directly apply the kernel trick in the dual and then obtain nonparametric models. Therefore, one does not need to formulate two different primal problems for the linear and nonlinear kernel respectively. In addition, experimental results are given to illustrate the performance of different loss functions.

Weighted Twin Support Vector Machines with Local Information and its application

A Twin Support Vector Machine (TWSVM), as a variant of a Multisurface Proximal Support Vector Machine via Generalized Eigenvalues (GEPSVM), attempts to improve the generalization of GEPSVM, whose s...

Weighted Twin Support Vector Machines with Local Information and its application

A Twin Support Vector Machine (TWSVM), as a variant of a Multisurface Proximal Support Vector Machine via Generalized Eigenvalues (GEPSVM), attempts to improve the generalization of GEPSVM, whose solution follows from solving two quadratic programming problems (QPPs), each of which is smaller than in a standard SVM. Unfortunately, the two QPPs still lead to rather high computational costs. Moreover, although TWSVM has better classification performance than GEPSVM, a major disadvantage is it fails to exploit the underlying correlation or similarity information between any pair of data points with the same labels that may be important for classification performance as much as possible. To mitigate the above deficiencies, in this paper, we propose a novel nonparallel plane classifier, called Weighted Twin Support Vector Machines with Local Information (WLTSVM). WLTSVM mines as much underlying similarity information within samples as possible. This method not only retains the superior characteristics of TWSVM, but also has its additional advantages: (1) comparable or better classification accuracy compared to SVM, GEPSVM and TWSVM; (2) taking motivation from standard SVM, the concept of support vectors is retained; (3) more efficient than TWSVM in terms of computational costs; and (4) only one penalty parameter is considered as opposed to two in TWSVM. Finally, experiments on both simulated and real problems confirm the effectiveness of our method.

1-Norm least squares twin support vector machines

During the last few years, nonparallel plane classifiers, such as Multisurface Proximal Support Vector Machine via Generalized Eigenvalues (GEPSVM), and Least Squares TWSVM (LSTSVM), have attracted much attention. However, there are not any modifications of them that have been presented to automatically select the input features. This motivates the rush towards new classifiers. In this paper, we develop a new nonparallel plane classifier, which is designed for automatically selecting the relevant features. We first introduce a Tikhonov regularization (TR) term that is usually used for regularizing least squares into the LSTSVM learning framework, and then convert this formulation to a linear programming (LP) problem. By minimizing an exterior penalty (EP) problem of the dual of the LP formulation and using a fast generalized Newton algorithm, our method yields very sparse solutions, such that it generates a classifier that depends on only a smaller number of input features. In other words, this approach is capable of suppressing input features. This makes the classifier easier to store and faster to compute in the classification phase. Lastly, experiments on both toy and real problems disclose the effectiveness of our method.

Localized twin SVM via convex minimization

Multisurface proximal support vector machine via generalized eigenvalues (GEPSVM), being an effective classification tool for supervised learning, tries to seek two nonparallel planes that are determined by solving two generalized eigenvalue problems (GEPs). The GEPs may lead to an instable classification performance, due to matrix singularity. Proximal support vector machine using local information (LIPSVM), as a variant of GEPSVM, attempts to avoid the above shortcoming through adopting a similar formulation to the Maximum Margin Criterion (MMC). The solution to an LIPSVM follows directly from solving two standard eigenvalue problems. Actually, an LIPSVM can be viewed as a reduced algorithm, because it uses the selectively generated points to train the classifier. A major advantage of an LIPSVM is that it is resistant to outliers. In this paper, following the geometric intuition of an LIPSVM, a novel multi-plane learning approach called Localized Twin SVM via Convex Minimization (LCTSVM) is proposed. This approach determines two nonparallel planes by solving two newly formed SVM-type problems. In addition to keeping the superior characteristics of an LIPSVM, an LCTSVM still has its additional edges: (1) it has similar or better classification capability compared to LIPSVM, TWSVM and LSTSVM; (2) each plane is generated from a quadratic programming problem (QPP) instead of a special convex difference optimization arising from an LIPSVM; (3) the solution can be reduced to solving two systems of linear equations, resulting in considerably lesser computational cost; and (4) it can find the global minimum. Experiments carried out on both toy and real-world problems disclose the effectiveness of an LCTSVM.