Large Quadratic Programming Problems Research Articles

Nonparametric Modelling of Ship Dynamics Using Puma Optimizer Algorithm-Optimized Twin Support Vector Regression

Ship dynamic models serve as the foundation for designing ship controllers, trajectory planning, and obstacle avoidance. Support vector regression (SVR) is a commonly used nonparametric modelling method for ship dynamics. Achieving high accuracy SVR models requires a substantial amount of training samples. Additionally, as the number of training samples increases, the computational efficiency for solving the quadratic programming problem (QPP) of SVR decreases. Ship controllers demand dynamic models with both high accuracy and computational efficiency. Therefore, to enhance the prediction accuracy and computational efficiency of SVR, this paper proposes a nonparametric modelling method based on twin SVR (TSVR). TSVR replaces a large QPP with a set of smaller QPPs, significantly enhancing generalizability and computational efficiency. To further improve the predictive accuracy of TSVR, the puma optimizer algorithm is employed to determine the optimal hyperparameters. The performance of the proposed method is validated using a Mariner class vessel. Gaussian white noise is introduced into the modelling data to simulate measurement error. The TSVR model accurately predicts various zigzag and turning circle manoeuvring motions under disturbance conditions, demonstrating its robustness and generalizability. Compared to the SVR model, the TSVR model achieves lower root mean square error and computational time, confirming its superior predictive accuracy and computational efficiency.

Robust twin depth support vector machine based on average depth

As one of the classical machine learning algorithms, twin support vector machine (TWSVM) can construct two nonparallel hyperplanes, which keeps the hyperplane close to points of one class, but far away from the other class. Being different from the classical SVM algorithm, TWSVM can reduce the computational complexity by replacing one large quadratic programming problems (QPPs) with a pair of smaller ones. Although compared to SVM classifier, TWSVM is four times faster, it is still insufficient in resisting noise or outliers. In this paper, we propose a twin depth support vector machine (TDSVM), a binary SVM classifier that considers the influence of depth when calculating the distance. A novel average depth is proposed and applied on TWSVM to construct a robust SVM framework, which can identify outliers in the dataset.By strengthening the center and weakening the edge, a better generalization performance is achieved, and the SRM principle is also implemented. TDSVM can be applied to any place where TWSVM can be applied, which is also useful for reducing the influence of outliers or noise on data and obtaining more robust results. Finally, the advantages of the method are verified theoretically and empirically by experiments. Experimental results on eight UCI datasets and one synthetic dataset demonstrate the effectiveness and robustness of TDSVM. The classification accuracy of TDSVM is better than other compared algorithms on almost every dataset, whether a certain percentage of Gaussian noise is introduced or not.

Transient stability assessment in bulk power grids using sequential minimal optimization based support vector machine with pinball loss

• A support vector machine with pinball loss is developed for TSA. • Quantile parameter is conducted to improve the robustness of TSA. • Sequential minimal optimization is adopted to accelerate the computational efficiency of TSA. Recent studies show that support vector machine is a promising approach for predicting the transient stability of power systems. However, most of the existing support vector machine methods suffer from excessive training time and low robustness, resulting in inefficiency for implementation. In this paper, a novel TSA method using sequential minimal optimization based support vector machine with pinball loss is developed. In the proposed approach, the system operation indices and projected energy function indices are selected to form the initial TSA feature set. Then, a quantile parameter is introduced to reduce the impact of measurement error by controlling the positions of the boundary samples between stable and unstable classes. Finally, a sequential minimal optimization is applied to decompose the large quadratic programming problem into a series of small quadratic programming problems to accelerate the computational efficiency. The cross-validation and grid search methods are introduced to improve the classification stability and accuracy. The IEEE 50-machine test system and the China Southern Power Grid are used to evaluate the performance of the proposed TSA method. Simulation results demonstrate that the proposed TSA method has high accuracy, robustness, and computational efficiency in bulk power grids.

Construct a robust least squares support vector machine based on L[formula omitted]-norm and L[formula omitted]-norm

Despite some Lp-norm LSSVMs own feature selection and prediction ability, they still suffer from two common issues. (i) They always ignore edge points because the L2-norm metric is used to measure the classification error of the training samples. The edge points are important in some practical application or on the datasets that are non-independent and identically distributed (non-i.i.d). (ii) They spend higher computational time and storage space for the large scale datasets. In order to solve the above two shortcomings while retaining the feature selection ability, we adopt L∞-norm to measure the classification error of training samples and still use Lp-norm (0<p<1) to measure the maximum margin between two parallel support planes, then obtain a novel LSSVM classifier, denoted as Lp-L∞-LSSVM. Our Lp-L∞-LSSVM owns three advantages: (1) L∞-norm on empirical risk ensures the effective recognition of edge points, thereby improving the robustness and generalization ability of the classifier. (2) Lp-norm on structural risk possess feature selection ability, whether for the linear or non-linear separable case and is suitable for the small samples size (SSS) problem. (3) Inspired by the sequential minimal optimization (SMO) algorithm, we designed an iterative heuristic algorithm by breaking the large quadratic programming problem (QPP) into a series of smallest possible QPPs, which can avoid high time-consuming. This algorithm not only ensures the convergence of optimum solution but also consumes lower computational time and storage space for large scale datasets. Finally, extensive numerical experiments once again verify the above opinions and show the outstanding classification performance and feature selection ability simultaneously.

An Online Robust Support Vector Regression for Data Streams

Since support vector regression (SVR) is a flexible regression algorithm, its computational complexity does not depend on the dimensionality of the input space, and it has excellent generalization capability. However, a central assumption with SVRs is that all the required data is available at the time of construction, which means these algorithms cannot be used with data streams. Incremental SVR has been offered as a potential solution, but its accuracy suffers with noise and learning speeds are slow. To overcome these two limitations, we propose a novel incremental regression algorithm, called online robust support vector regression (ORSVR). ORSVR solves nonparallel bound functions simultaneously. Hence, the large quadratic programming problem (QPP) in classical v-SVR are decomposed into two smaller QPPs. An incremental learning algorithm then solves each QPP step-by-step. The results of a series of comparative experiments demonstrate that the ORSVR algorithm efficiently solves regression problems in data streams, with or without noise, and speeds up the learning process.

Sparse pinball twin support vector machines

The original twin support vector machine (TWSVM) formulation works by solving two smaller quadratic programming problems (QPPs) as compared to the traditional hinge-loss SVM (C-SVM) which solves a single large QPP — this makes the TWSVM training and testing process faster than the C-SVM. However, these TWSVM problems are based on the hinge-loss function and, hence, are sensitive to feature noise and unstable for re-sampling. The pinball-loss function, on the other hand, maximizes quantile distances which grants noise insensitivity but this comes at the cost of losing sparsity by penalizing correctly classified samples as well. To overcome the limitations of TWSVM, we propose a novel sparse pinball twin support vector machines (SPTWSVM) based on the ϵ-insensitive zone pinball loss function to rid the original TWSVM of its noise insensitivity and ensure that the resulting TWSVM problems retain sparsity which makes computations relating to predictions just as fast as the original TWSVM. We further investigate the properties of our SPTWSVM including sparsity, noise insensitivity, and time complexity. Exhaustive testing on several benchmark datasets demonstrates that our SPTWSVM is noise insensitive, retains sparsity and, in most cases, outperforms the results obtained by the original TWSVM.

Generalized eigenvalue proximal support vector regressor for the simultaneous learning of a function and its derivatives

Generalized eigenvalue proximal support vector regressor (GEPSVR) determines a pair of \(\epsilon\)-insensitive bounding regressors by solving a pair of generalized eigenvalue problem. On the lines of GEPSVR, in this paper we propose a novel regressor for the simultaneous learning of a function and its derivatives, termed as GEPSVR of a Function and its Derivatives. The proposed method is fast as it requires the solution of a pair of generalized eigenvalue problems as compared to the solution of a large Quadratic Programming Problem required in other existing approaches. The experiment results on several benchmark functions of more than one variable proves the efficacy of our proposed method.

Nonparallel least square support vector machine for classification

Nonparallel support vector machine based on one optimization problem (NSVMOOP) aims at finding two nonparallel hyper-planes by maximizing the intersection angle of their normal vectors w 1 and w 2. As maximum intersection angle preserves both compactness and separation of data, NSVMOOP yields good forecasting accuracy. However, as it solves one large quadratic programming problem (QPP), it costs high running time. In order to improve its learning speed, a novel nonparallel least square support vector machine (NLSSVM) is proposed in this paper. NLSSVM solves a linear system of equations instead of solving one large QPP. As both intersection angle and least square version are applied on our NLSSVM, it performs better generalization performance than other algorithms. Experimental results on twenty benchmark datasets demonstrate its validity.

An efficient implicit regularized Lagrangian twin support vector regression

Twin support vector regression (TSVR) and Lagrangian TSVR (LTSVR) satisfy only empirical risk minimization principle. Moreover, the matrices in their formulations are always positive semi-definite. To overcome these problems, we propose an efficient implicit Lagrangian formulation for the dual regularized twin support vector regression, called IRLTSVR for short. By introducing a regularization term to each objective function, the optimization problems in our IRLTSVR are positive definite and implement the structural risk minimization principle. Moreover, the 1-norm of the vector of slack variable is replaced with 2-norm to make the objective functions strongly convex. Our IRLTSVR solves two systems of linear equations instead of solving two quadratic programming problems (QPPs) in TSVR and one large QPP in SVR, which makes the learning speed of IRLTSVR faster than TSVR and SVR. Particularly, we compare three implementations of IRLTSVR with existing approaches. Computational results on several synthetic and real-world benchmark datasets clearly indicate the effectiveness and applicability of the IRLTSVR in comparison to SVR, TSVR and LTSVR.

A regularization on Lagrangian twin support vector regression

Twin support vector regression (TSVR), Lagrangian TSVR (LTSVR) and $$\epsilon$$ -TSVR obtain good generalization and faster computational speed by solving a pair of smaller sized quadratic programming problems (QPPs) than a single large QPP in support vector regression (SVR). In this paper, a simple and linearly convergent Lagrangian support vector machine algorithm for the dual of the $$\epsilon$$ -TSVR is proposed. The contributions of our formulation are as follows: (1) we consider the square of the 2-norm of the vector of slack variables instead of the usual 1-norm to make the objective functions strongly convex. (2) We are solving regression problem with just two systems of linear equations as opposed to solving two QPPs in $$\epsilon$$ -TSVR and TSVR or one large QPP in SVR, which leads to extremely simple and fast algorithm. (3) One significant advantage of our proposed method is the implementation of structural risk minimization principle. However, only empirical risk is considered in the primal problems of TSVR and LTSVR due to its complex structure and thus may incur overfitting and suboptimal in some cases. (4) The experimental results on several artificial and benchmark datasets show the effectiveness of our proposed formulation.

Support vector machine for diagnosis cancer disease: A comparative study

Support vector machine has become an increasingly popular tool for machine learning tasks involving classification, regression or novelty detection. Training a support vector machine requires the solution of a very large quadratic programming problem. Traditional optimization methods cannot be directly applied due to memory restrictions. Up to now, several approaches exist for circumventing the above shortcomings and work well. Another learning algorithm, particle swarm optimization, Quantum-behave Particle Swarm for training SVM is introduced. Another approach named least square support vector machine (LSSVM) and active set strategy are introduced. The obtained results by these methods are tested on a breast cancer dataset and compared with the exact solution model problem.

New approach to training support vector machine

Abstract Support vector machine has become an increasingly popular tool for machine learning tasks involving classification, regression or novelty detection. Training a support vector machine requires the solution of a very large quadratic programming problem. Traditional optimization methods cannot be directly applied due to memory restrictions. Up to now, several approaches exist for circumventing the above shortcomings and work well. Another learning algorithm, particle swarm optimization, for training SVM is introduted. The method is tested on UCI datasets.

An extended Lagrangian support vector machine for classifications*

Abstract Lagrangian support vector machine (LSVM) cannot solve large problems for nonlinear kernel classifiers. In order to extend the LSVM to solvevery large problems, an extended Lagrangian support vector machine (ELSVM) for classifications based on LSVM and SVMlight is presented in this paper. Our idea for the ELSVM is to divide a large quadratic programming problem into a series of subproblems with small size and to solve them via LSVM. Since the LSVM can solve small and medium problems for nonlinear kernel classifiers, the proposed ELSVM can be used to handle large problems very efficiently. Numerical experiments on different types of problems are performed to demonstrate the high efficiency of the ELSVM.

A Proportioning Based Algorithm with Rate of Convergence for Bound Constrained Quadratic Programming

The proportioning algorithm with projections turned out to be an efficient algorithm for iterative solution of large quadratic programming problems with simple bounds and box constraints. Important features of this active set based algorithm are the adaptive precision control in the solution of auxiliary linear problems and capability to add or remove many indices from the active set in one step. In this paper a modification of the algorithm is presented that enables to find its rate of convergence in terms of the spectral condition number of the Hessian matrix and avoid any backtracking. The modified algorithm is shown to preserve the finite termination property of the original algorithm for problems that are not dual degenerate.

On the Solution of Large Quadratic Programming Problems with Bound Constraints

An algorithm is proposed that uses the conjugate gradient method to explore the face of the feasible region defined by the current iterate, and the gradient projection method to move to a different face. It is proved that for strictly convex problems the algorithm converges to the solution, and that if the solution is nondegenerate, then the algorithm terminates at the solution in a finite number of steps. Numerical results are presented for the obstacle problem, the elastic-plastic torsion problem, and the journal bearing problems. On a selection of these problems with dimensions ranging from 5000 to 15,000, the algorithm determines the solution in fewer than 15 iterations, and with a small number of function-gradient evaluations and Hessian-vector products per iteration.

A decomposition technique for an optimal control problem with "PQDZ" cost and bounded controls

In this note we present a solution method for a discrete-time linear optimal control problem where the controls are bounded and both the states and controls have asymmetric costs with dead zones containing the nominal values. This model generalizes the well-known optimal control model for a linear system with symmetric (quadratic) objective functional. We transform the problem into an equivalent large quadratic programming problem with equality and inequality, constraints and decompose it into N simpler subproblems, each with very easily computed optimal solutions. Using a two-level approach suggested by Lasdon and Schoeffler [11] the optimal solution to the overall problem is obtained.