Support Vector Machine Formulation Research Articles

Kernel Antenna Array Processing

We introduce two support vector machine (SVM)-based approaches for solving antenna problems such as beamforming, sidelobe suppression, and maximization of the signal-to-noise ratio. A basic introduction to SVM optimization is provided and a complex nonlinear SVM formulation developed to handle antenna array processing in space and time. The new optimization formulation is compared with both the minimum mean square error and the minimum variance distortionless response methods. Several examples are included to show the performance of the new approaches

Incremental Support Vector Machine Framework for Visual Sensor Networks

Motivated by the emerging requirements of surveillance networks, we present in this paper an incremental multiclassification support vector machine (SVM) technique as a new framework for action classification based on real-time multivideo collected by homogeneous sites. The technique is based on an adaptation of least square SVM (LS-SVM) formulation but extends beyond the static image-based learning of current SVM methodologies. In applying the technique, an initial supervised offline learning phase is followed by a visual behavior data acquisition and an online learning phase during which the cluster head performs an ensemble of model aggregations based on the sensor nodes inputs. The cluster head then selectively switches on designated sensor nodes for future incremental learning. Combining sensor data offers an improvement over single camera sensing especially when the latter has an occluded view of the target object. The optimization involved alleviates the burdens of power consumption and communication bandwidth requirements. The resulting misclassification error rate, the iterative error reduction rate of the proposed incremental learning, and the decision fusion technique prove its validity when applied to visual sensor networks. Furthermore, the enabled online learning allows an adaptive domain knowledge insertion and offers the advantage of reducing both the model training time and the information storage requirements of the overall system which makes it even more attractive for distributed sensor networks communication.

Support vector machine via nonlinear rescaling method

In this paper we construct the linear support vector machine (SVM) based on the nonlinear rescaling (NR) methodology (see [Polyak in Math Program 54:177–222, 1992; Polyak in Math Program Ser A 92:197–235, 2002; Polyak and Teboulle in Math Program 76:265–284, 1997] and references therein). The formulation of the linear SVM based on the NR method leads to an algorithm which reduces the number of support vectors without compromising the classification performance compared to the linear soft-margin SVM formulation. The NR algorithm computes both the primal and the dual approximation at each step. The dual variables associated with the given data-set provide important information about each data point and play the key role in selecting the set of support vectors. Experimental results on ten benchmark classification problems show that the NR formulation is feasible. The quality of discrimination, in most instances, is comparable to the linear soft-margin SVM while the number of support vectors in several instances were substantially reduced.

Support vector machines for robust channel estimation in OFDM

A new support vector machine (SVM) algorithm for coherent robust demodulation in orthogonal frequency-division multiplexing (OFDM) systems is proposed. We present a complex regression SVM formulation specifically adapted to a pilots-based OFDM signal. This novel proposal provides a simpler scheme than an SVM classification method. The feasibility of our approach is substantiated by computer simulation results obtained for IEEE 802.16 broadband fixed wireless channel models. These experiments allow to scrutinize the performance of the OFDM-SVM system and the suitability of the epsiv-Huber cost function, in the presence of non-Gaussian impulse noise interfering with OFDM pilot symbols

Support Vector Machines for Environmental Informatics: Application to Modelling the Nitrogen Removal Processes in Wastewater Treatment Systems

In order to meet the new stringent environmental regulations, it is necessary to investigate the adaptive and optimal control strategies for the biological wastewater treatment processes. Nitrogen removal is one of the essential concerns in wastewater treatment. Nitrogen removal is a nonlinear, dynamic, and time variant complex process as complicated activities of microbial metabolism are involved. The mechanistic models for nitrogen removal are complicated and still uncertain to some extent. A new machine learning approach, Support Vector Machine (SVM) was proposed as black-box modeling technique to model the biological wastewater treatment processes. LS-SVM, a simplified formulation of SVM, has been applied in this study to predict the concentration of nitrate and nitrite (NO) in the Mixed Liquor (ML) of wastewater treatment plant. Nonlinear Autoregressive model with Exogenous inputs (NARX model) can be employed with LS-SVM to extract useful information and improve the prediction performance. In this paper, the premium wastewater treatment plant simulation and optimization software, GPS-X, is used to create virtual plant layout and simulated data. The simulation results indicate that the proposed method has good generalization performance, especially when the input is fluctuated without a usual pattern. We conclude that LS-SVM with NARX modelling could be used as an alternative approach to predict the behaviour of wastewater treatment systems by further studying some essential issues such as the tuning of memory order and training data size.

Dual unification of bi-class support vector machine formulations

Support vector machine (SVM) theory was originally developed on the basis of a linearly separable binary classification problem, and other approaches have been later introduced for this problem. In this paper it is demonstrated that all these approaches admit the same dual problem formulation in the linearly separable case and that all the solutions are equivalent. For the non-linearly separable case, all the approaches can also be formulated as a unique dual optimization problem, however, their solutions are not equivalent. Discussions and remarks in the article point to an in-depth comparison between SVM formulations and associated parameters.

An Improved Conjugate Gradient Scheme to the Solution of Least Squares SVM

The least square support vector machines (LS-SVM) formulation corresponds to the solution of a linear system of equations. Several approaches to its numerical solutions have been proposed in the literature. In this letter, we propose an improved method to the numerical solution of LS-SVM and show that the problem can be solved using one reduced system of linear equations. Compared with the existing algorithm for LS-SVM, the approach used in this letter is about twice as efficient. Numerical results using the proposed method are provided for comparisons with other existing algorithms.

Intrusion Detection in Unlabeled Data with Quarter-sphere Support Vector Machines

The anomaly detection methods are receiving growing attention in the intrusion detection community. The two main reasons for this are their ability to handle large volumes of unlabeled data and to detect previously unknown attacks. In this contribution we investigate the application of a modern machine learning technique – one-class Support Vector Machines (SVM) – for anomaly detection in unlabeled data. We propose a novel formulation of this technique which is particularly suited for the data typical for intrusion detection systems. Our evaluation on the well-known KDDCup dataset demonstrates a significant improvement over previous formulations of the one-class SVM.

Intrusion detection in unlabeled data with quarter-sphere Support Vector Machines

Practical application of data mining and machine learning techniques to intrusion detection is often hindered by the difficulty to produce clean data for the training. To address this problem a geometric framework for unsupervised anomaly detection has been recently proposed. In this framework, the data is mapped into a feature space, and anomalies are detected as the entries in sparsely populated regions. In this contribution we propose a novel formulation of a one-class Support Vector Machine (SVM) specially designed for typical IDS data features. The key idea of our ”quarter-sphere” algorithm is to encompass the data with a hypersphere anchored at the center of mass of the data in feature space. The proposed method and its behavior on varying percentages of attacks in the data is evaluated on the KDDCup 1999 dataset.

A support vector machine formulation to pca analysis and its kernel version

In this paper, we present a simple and straightforward primal-dual support vector machine formulation to the problem of principal component analysis (PCA) in dual variables. By considering a mapping to a high-dimensional feature space and application of the kernel trick (Mercer theorem), kernel PCA is obtained as introduced by Scholkopf et al. (2002). While least squares support vector machine classifiers have a natural link with the kernel Fisher discriminant analysis (minimizing the within class scatter around targets +1 and -1), for PCA analysis one can take the interpretation of a one-class modeling problem with zero target value around which one maximizes the variance. The score variables are interpreted as error variables within the problem formulation. In this way primal-dual constrained optimization problem interpretations to the linear and kernel PCA analysis are obtained in a similar style as for least square-support vector machine classifiers.

SMO algorithm for least-squares SVM formulations.

This article extends the well-known SMO algorithm of support vector machines (SVMs) to least-squares SVM formulations that include LS-SVM classification, kernel ridge regression, and a particular form of regularized kernel Fisher discriminant. The algorithm is shown to be asymptotically convergent. It is also extremely easy to implement. Computational experiments show that the algorithm is fast and scales efficiently (quadratically) as a function of the number of examples.

EXTRACTING FEATURE SUBSPACE FOR KERNEL BASED LINEAR PROGRAMMING SUPPORT VECTOR MACHINES

We propose linear programming formulations of support vector machines (SVM). Unlike standard SVMs which use quadratic programs, our approach explores a fairly small dimensional subspace of a feature space to construct the nonlinear discriminator. This allows us to obtain the discriminator by solving a smaller sized linear program. We demonstrate that an orthonormal basis of the subspace can be implicitly treated by eigenvectors of the Gram matrix defined by the associated kernel function. When the number of given data points is very large, we construct a subspace by random sampling of data points. Numerical experiments indicate that the subspace generated by less than 2% of the entire training data points achieves reasonable performance for a fairly large instance with 60000 data points.

The Equivalence of Support Vector Machine and Regularization Neural Networks

We show in this brief paper the equivalence of the support vector machine and regularization neural networks. We prove both implication sides of the equivalence in a generally applicable way. The novelty lies in the effective construction of the regularization operator corresponding to a given support vector machine formulation. We give also a short introductory description of both neural network approximation frameworks.

Interior-Point Methods for Massive Support Vector Machines

We investigate the use of interior-point methods for solving quadratic programming problems with a small number of linear constraints, where the quadratic term consists of a low-rank update to a positive semidefinite matrix. Several formulations of the support vector machine fit into this category. An interesting feature of these particular problems is the volume of data, which can lead to quadratic programs with between 10 and 100 million variables and, if written explicitly, a dense Q matrix. Our code is based on OOQP, an object-oriented interior-point code, with the linear algebra specialized for the support vector machine application. For the targeted massive problems, all of the data is stored out of core and we overlap computation and input/output to reduce overhead. Results are reported for several linear support vector machine formulations demonstrating that the method is reliable and scalable.

Formulations of Support Vector Machines: A Note from an Optimization Point of View

In this article, we discuss issues about formulations of support vector machines (SVM) from an optimization point of view. First, SVMs map training data into a higher- (maybe infinite-) dimensional space. Currently primal and dual formulations of SVM are derived in the finite dimensional space and readily extend to the infinite-dimensional space. We rigorously discuss the primal-dual relation in the infinite-dimensional spaces. Second, SVM formulations contain penalty terms, which are different from unconstrained penalty functions in optimization. Traditionally unconstrained penalty functions approximate a constrained problem as the penalty parameter increases. We are interested in similar properties for SVM formulations. For two of the most popular SVM formulations, we show that one enjoys properties of exact penalty functions, but the other is only like traditional penalty functions, which converge when the penalty parameter goes to infinity.

The analysis of decomposition methods for support vector machines

The support vector machine (SVM) is a new and promising technique for pattern recognition. It requires the solution of a large dense quadratic programming problem. Traditional optimization methods cannot be directly applied due to memory restrictions. Up to now, very few methods can handle the memory problem and an important one is the "decomposition method." However, there is no convergence proof so far. In this paper, we connect this method to projected gradient methods and provide theoretical proofs for a version of decomposition methods. An extension to bound-constrained formulation of SVM is also provided.We then show that this convergence proof is valid for general decomposition methods if their working set selection meets a simple requirement.

PREFACE: Guest Editorial – New Trends in Neural Network Implementations

In the last decade, the efforts of the neural network research community have been concentrated on investigating the abilities and limits of neural networks in general. This focus was both necessary and timely. Before trying to use a tool, we must understand its possibilities and limits. A necessary and essential step in continuing the diffusion of this paradigm in our day by day use is their hardware implementation. The ability of the neural systems to be implemented in hardware is crucial because the normal requirements for many applications include: i) a small volume, ii) a reduced weight, iii) a high resistance to shocks, vibrations and adverse environmental conditions and iv) a high execution speed. Furthermore, the hardware implementation is by far the most cost effective solution for large scale use. This special issue of the International Journal of Neural Systems is dedicated to New Trends in Neural Network Implementations. The 8 papers selected went through a careful review process in order to ensure the high quality of the issue. Approximatively 50% of the manuscripts submitted were accepted for publication. Each paper was reviewed by 3 independent reviewers and the editor of this issue. After this first round of reviews, the accepted papers went back to the authors together with the comments of the reviewers. The final versions of the papers were reviewed again before publication for compliance with the reviewers' requests. The first paper, by Anguita, Boni and Ridella present some algorithms and architectures for a digital VLSI implementation of support vector machines (SVM). The structural risk minimization approach with its most common implementation in the form of SVMs has shown to be able to provide very good performance in various application. This paper discusses the main aspects concerning the SVM learning in hardware. Issues like the effects of fixed point computation and the storage are addressed in this context. The second paper, by Waldemark, Millberg, Lindblad, Waldemark and Becanovic presents a biologically inspired neural network for image analysis applications. The Pulse Coupled Neural Network (PCNN) is an image pre-processor that can be combined with other application specific parts to form a complete system. This paper presents a VHDL implementation of the PCNN targeting field-programmable gate arrays (FPGA). The next paper is this issue deals with a similar type of image processing application. The paper by Serrano-Gotarredona, Andreou and Linares-Barranco presents an architecture for the realization of real-time edge extraction filtering in an Address-Event-Representation (AER) vision system. In fact, the approach presented here is valid for any 2-dimensional filtering system using a kernel in which the x-axis and y-axis can be separated. This paper presents a circuit implementation using MOS transistors operated in weak inversion. The proposed architecture is representative of the success of the Boundary-Contour-System and Feature-Contour-System vision model developed over the years by the research group lead by Stephen Grossberg at Boston University. The paper by Wilamowski, Binfet and Kaynak presents two neural networks designed in 1.2/xm CMOS technology. Interesting aspects addressed by this paper include compensating for imperfect activation functions and using quantized weights with limited precision. The paper by Duong presents a so called VLSI-friendly algorithm i.e. an algorithm that has certain properties that allow it to be implemented efficiently in VLSI hardware. The Cascade Error Projection (CEP) is a constructive algorithm inspired by Fahlman's Cascade Correlation. The paper includes a convergence analysis

Massive data discrimination via linear support vector machines

A linear support vector machine formulation is used to generate a fast, finitely-terminating linear-programming algorithm for discriminating between two massive sets in n-dimen-sional space, where the number of points can be orders of magnitude larger than n. The algorithm creates a succession of sufficiently small linear programs that separate chunks of the data at a time. The key idea is that a small number of support vectors, corresponding to linear programming constraints with positive dual variables, are carried over between the successive small linear programs, each of which containing a chunk of the data. We prove that this procedure is monotonic and terminates in a finite number of steps at an exact solution that leads to an optimal separating plane for the entire dataset. Numerical results on fully dense publicly available datasets, numbering 20,000 to 1 million points in 32-dimensional space, confirm the theoretical results and demonstrate the ability to handle very large problems

A36 Analysis of long term adverse experience data using the Weibull model

Machine learning models are widely used for anomaly detection in network traffic. Effective transformation of the raw traffic data into mathematical expressions and hyper-parameter adjustment are two important steps before training the machine learning classifier, which is used to predict whether the unknown traffic is normal or abnormal. In this paper, a novel model SVM-L is proposed for anomaly detection in network traffic. In particular, raw URLs are treated as natural language, and then transformed into mathematical vectors via statistical laws and natural language processing technique. They are used as the training data for the traffic classifier, the kernel Support Vector Machine (SVM). Based on the idea of the dual formulation of kernel SVM and Linear Discriminant Analysis (LDA), we propose an optimization model to adjust the hyper-parameter of the classifier. The corresponding problem is simply one-dimensional, and is easily solved by the golden section method. Numerical tests indicate that the proposed model achieves more than 99% accuracy on all tested datasets, and outperforms the state of the arts in terms of standard evaluation measurements.