Lagrangian Support Vector Regression Research Articles

A Hybrid Model Based on Grey Wolf Optimizer and Lagrangian Support Vector Regression for European Natural Gas Consumption Forecasting

Natural gas plays an important role in industry as a clean energy, with the intensification of the Russia-Ukraine war, there is a large-scale energy shortage in Europe, and the natural gas supply in Europe has a natural gas crisis due to the cut-off of the Nord Stream No.1 pipeline. Therefore, it is necessary to accurately predict the consumption of natural gas. In order to fulfill this requirement, this paper uses the Lagrangian Support Vector Regression model with Sorensen kernel based on the Nonlinear Auto-Regressive model and Grey Wolf Optimizer for 5-step forecasting of monthly natural gas consumption in all European countries. Under three time lags, comparing the 5-step predict results of GWO-LSVR with SVR, RF, LightGBM, XGBoost, and MLP, those five models’ hyperparameters also optimized by GWO, it found that GWO-LSVR has smallest MAPE in almost all cases, and the numerical results of MAPE generated by GWO-LSVR is from 5.844% to 11.622%, the smaller the forecasting step size, the better the effect. Moreover, compares the difference of GWO and WOA, it is found that GWO can obtained better model hyperparameters and smaller MAPE results. To sum up, the proposed GWO-LSVR model has strong generalization performance and robustness, and is a reliable natural gas consumption prediction model.

Efficient implicit Lagrangian twin parametric insensitive support vector regression via unconstrained minimization problems

In this paper, an efficient implicit Lagrangian twin parametric insensitive support vector regression is proposed which leads to a pair of unconstrained minimization problems, motivated by the works on twin parametric insensitive support vector regression (Peng: Neurocomputing. 79, 26–38, 2012), and Lagrangian twin support vector regression (Balasundaram and Tanveer: Neural Comput. Applic. 22(1), 257–267, 2013). Since its objective function is strongly convex, piece-wise quadratic and differentiable, it can be solved by gradient-based iterative methods. Notice that its objective function having non-smooth ‘plus’ function, so one can consider either generalized Hessian, or smooth approximation function to replace the ‘plus’ function and further apply the simple Newton-Armijo step size algorithm. These algorithms can be easily implemented in MATLAB and do not require any optimization toolbox. The advantage of this method is that proposed algorithms take less training time and can deal with data having heteroscedastic noise structure. To demonstrate the effectiveness of the proposed method, computational results are obtained on synthetic and real-world datasets which clearly show comparable generalization performance and improved learning speed in accordance with support vector regression, twin support vector regression, and twin parametric insensitive support vector regression.

On a new approach for Lagrangian support vector regression

In this paper, a simplification of the necessary and sufficient Karush–Kuhn–Tucker (KKT) optimality conditions for the Lagrangian support vector regression in 2-norm resulting a naive root finding problem of a system of equations in m variables is proposed, where m is the number of training examples. However, since the resulting system of equations contains terms having the nonsmooth “plus” function, two approaches are considered: (i) the problem is reformulated into an equivalent absolute value equation problem and solved by functional iterative and generalized Newton methods and (ii) solve the original root finding problem by generalized Newton method. The proofs of convergence and pseudo-codes of the iterative methods are given. Numerical experiments performed on a number of synthetic and real-world benchmark datasets showing similar generalization performance with much faster learning speed in comparison with support vector regression (SVR) and as good as performance in comparison with the least squares SVR, unconstrained Lagrangian SVR proposed in Balasundaram et al. (Neural Netw 51:67–79, 2014), twin SVR and extreme learning machine clearly indicate the effectiveness and suitability of the proposed problem formulation solved by functional iterative and generalized Newton methods.

ESTABLISHMENT OF EFFECTIVE METAMODELS FOR SEAKEEPING PERFORMANCE IN MULTIDISCIPLINARY SHIP DESIGN OPTIMIZATION

Ship design is a complex multidisciplinary optimization process to determine configuration variables that satisfy a set of mission requirements. Unfortunately, high fidelity commercial software for the ship performance estimation such as Computational Fluid Dynamics (CFD) and Finite Element Analysis (FEA) is computationally expensive and time consuming to execute and deters the ship designer’s ability to explore larger range of optimization solutions. In this paper, the Latin Hypercube Design was used to select the sample data for covering the design space. The percentage of downtime, a comprehensive seakeeping evaluation index, was also used to evaluate the seakeeping performance within the short-term and long-term wave distribution in the process of Multidisciplinary Design Optimization (MDO). The five motions of ship seakeeping performance contained roll, pitch, yaw, sway and heave. Particularly, a new effective approximation modelling technique-Single-Parameter Lagrangian Support Vector Regression (SPL-SVR) was investigated to construct ship seakeeping metamodels to facilitate the application of MDO. By considering the effects of two ship speeds, the established metamedels of ship seakeeping performance for the short-term percentage of downtime are satisfactory for seakeeping predictions during the conceptual design stage; thus, the new approximation algorithm provides an optimal and cost-effective solution for constructing the metamodels in MDO process.

A new approach for training Lagrangian support vector regression

In this paper, a novel root finding problem for the Lagrangian support vector regression in 2-norm (LSVR) is formulated in which the number of unknowns becomes the number of training examples. Further, it is proposed to solve it by functional iterative and Newton methods. Under sufficient conditions, we proved their linear rate of convergence. Experiments are performed on a number of synthetic and real-world benchmark datasets, and their results are compared with support vector regression (SVR) and its variants such as least squares SVR and LSVR. Similar generalization performance with improved or comparable learning speed to SVR and its variants demonstrates the usefulness of the proposed formulation solved by the iterative methods.

LWT- QR decomposition based robust and efficient image watermarking scheme using Lagrangian SVR

In this paper, an efficient and robust image watermarking scheme based on lifting wavelet transform (LWT) and QR decomposition using Lagrangian support vector regression (LSVR) is presented. After performing one level decomposition of host image using LWT, the low frequency subband is divided into 4?×?4 non-overlapping blocks. Based on the correlation property of lifting wavelet coefficients, each selected block is followed by QR decomposition. The significant element of first row of R matrix of each block is set as target to LSVR for embedding the watermark. The remaining elements (called feature vector) of upper triangular matrix R act as input to LSVR. The security of the watermark is achieved by applying Arnold transformation to original watermark to get its scrambled image. This scrambled image is embedded into the output (predicted value) of LSVR compared with the target value using optimal scaling factor to reduce the tradeoff between imperceptibility and robustness. Experimental results show that proposed scheme not only efficient in terms of computational cost and memory requirement but also achieve good imperceptibility and robustness against image processing operations compared to the state-of-art techniques.

A robust and efficient image watermarking scheme based on Lagrangian SVR and lifting wavelet transform

In this paper, an efficient image watermarking scheme based on Lagrangian support vector regression (LSVR) and lifting wavelet transform (LWT) is proposed to balance the trade-off between imperceptibility and robustness. LWT is faster and efficient implementation of traditional wavelet transform and LSVR has faster learning speed and high generalization capability compared to classical support vector regression. Combination of LWT and LSVR based image watermarking not only show imperceptibility and robustness but also reduces the time complexity. First the image is decomposed into four subbands using one level LWT and low frequency subband (approximate image) is partitioned into non overlapping blocks. Selected blocks based on correlation of wavelet coefficients are used to embed the binary watermark logo. For each selected block, significant wavelet coefficient is used as target to LSVR and its neighboring coefficients (feature vector) act as input to LSVR. The security of the watermark is achieved by applying Arnold transformation to original watermark. The scrambled watermark bit is embedded by comparing the output (predicted value) of LSVR of each selected block and the actual target value. The good learning capability and high generalization property of LSVR against noisy datasets provides the successful extraction of watermark against several image processing attacks: median filtering, average filtering, addition of Gaussian noise, salt and pepper noise, contrast enhancement, blurring, scaling, cropping and rotation. Experimental results show that the proposed scheme gives significant improvement in imperceptibility and robustness compared to the state-of-art techniques.

Robust image watermarking scheme in lifting wavelet domain using GA-LSVR hybridization

This paper presents an imperceptible, robust, secure and efficient image watermarking scheme in lifting wavelet domain using combination of genetic algorithm (GA) and Lagrangian support vector regression (LSVR). First, four subbands low–low (LL), low–high (LH), high–low (HL) and high–high (HH) are obtained by decomposing the host image from spatial domain to frequency domain using one level lifting wavelet transform. Second, the approximate image (LL subband) is divided into non overlapping blocks and the selected blocks based on the fuzzy entropy are used to embed the binary watermark. Third, based on the correlation property of each transformed selected block, significant lifting wavelet coefficient act as target to LSVR and its neighboring coefficients (called feature vector) are set as input to LSVR to find optimal regression function. This optimal regression function is used to embed and extract the scrambled watermark. In the proposed scheme, GA is used to solve the problem of optimal watermark embedding strength, based on the noise sensitivity of each selected block, in order to increase the imperceptibility of the watermark. Due to the good learning capability and high generalization property of LSVR against noisy datasets, high degree of robustness is achieved and is well suited for copyright protection applications. Experimental results on standard and real world images show that proposed scheme not only efficient in terms of computational cost and memory requirement but also achieve good imperceptibility and robustness against geometric and non geometric attacks like JPEG compression, median filtering, average filtering, addition of noise, sharpening, scaling, cropping and rotation compared with the state-of-art techniques.

Gray scale image watermarking using fuzzy entropy and Lagrangian support vector regression in DCT domain

A new image watermarking scheme based on fuzzy entropy and Lagrangian support vector regression (LSVR) in discrete cosine transform (DCT) domain is proposed in this paper. Fuzzy entropy is used to extract smooth non-overlapping blocks of an image followed by DCT to each selected block to transform it from spatial domain to frequency domain. Then a feature vector of low frequency DCT coefficients of each block act as an input to LSVR. The output (predicted value) of trained LSVR is used to embed the binary watermark by comparing it with the target value of the feature vector. As fuzzy entropy is able to discriminate data distribution under noise corrupted and redundant conditions, feature extraction is more robust against various attacks. The robustness of the proposed scheme is verified by performing various types of image processing operations which include geometric and non-geometric attacks. It is evident from experimental results that with the help of randomness measured of each image block using fuzzy entropy, good learning ability and high generalisation property of LSVR have made the proposed scheme more imperceptible and robust against different types of image processing operations.

Lagrangian support vector regression via unconstrained convex minimization

In this paper, a simple reformulation of the Lagrangian dual of the 2-norm support vector regression (SVR) is proposed as an unconstrained minimization problem. This formulation has the advantage that its objective function is strongly convex and further having only m variables, where m is the number of input data points. The proposed unconstrained Lagrangian SVR (ULSVR) is solvable by computing the zeros of its gradient. However, since its objective function contains the non-smooth ‘plus’ function, two approaches are followed to solve the proposed optimization problem: (i) by introducing a smooth approximation, generate a slightly modified unconstrained minimization problem and solve it; (ii) solve the problem directly by applying generalized derivative. Computational results obtained on a number of synthetic and real-world benchmark datasets showing similar generalization performance with much faster learning speed in accordance with the conventional SVR and training time very close to least squares SVR clearly indicate the superiority of ULSVR solved by smooth and generalized derivative approaches.

Finite Newton algorithm for Lagrangian support vector regression

Lagrangian Support Vector Regression(SVR) is an effective algorithm and its solution is obtained by taking the inverse of a matrix of order equaling the number of samples plus one,but needs many times to terminate from a starting point.This paper proposed a finite Armijo-Newton algorithm solving the Lagrangian SVR's optimization problem.A solution was obtained by solving a system of linear equations at a finite number of times rather than solving a quadratic programming problem.The proposed method has the advantage that the result optimization problem is solved with global convergence and finite-step termination.The experimental results on several benchmark datasets indicate that the proposed algorithm is fast,and shows good generalization performance.

Finite Newton method for implicit Lagrangian support vector regression

In this paper a finite Newton iterative method of solution for solving the implicit Lagrangian Support Vector Regression (SVR) formulation has been proposed. Unlike solving a quadratic programming problem for the case of the standard SVR the solution

On Lagrangian support vector regression

Prediction by regression is an important method of solution for forecasting. In this paper an iterative Lagrangian support vector machine algorithm for regression problems has been proposed. The method has the advantage that its solution is obtained by taking the inverse of a matrix of order equals to the number of input samples at the beginning of the iteration rather than solving a quadratic optimization problem. The algorithm converges from any starting point and does not need any optimization packages. Numerical experiments have been performed on Bodyfat and a number of important time series datasets of interest. The results obtained are in close agreement with the exact solution of the problems considered clearly demonstrates the effectiveness of the proposed method.