Kernel Regularization Research Articles

The Cobb-Douglas Learning Machine

In this paper, we propose a novel machine learning approach based on robust optimization. Our proposal defines the task of maximizing the two class accuracies of a binary classification problem as a Cobb-Douglas function. This function is well known in production economics and is used to model the relationship between two or more inputs as well as the quantity produced by those inputs. A robust optimization problem is defined to construct the decision function. The goal of the model is to classify each training pattern correctly, up to a given class accuracy, even for the worst possible data distribution. We demonstrate the theoretical advantages of the Cobb-Douglas function in terms of the properties of the resulting second-order cone programming problem. Important extensions are proposed and discussed, including the use of kernel functions and regularization. Experiments performed on several classification datasets confirm these advantages, leading to the best average performance in comparison to various alternative classifiers.

Research on Modeling of Underground Path Loss Prediction Based on SA-SVM

In view of the serious refraction and reflection of radio wave transmission in the complex environment of coal mines, serious multipath fading occurs, which affects the performance of the communication system. Therefore, we presents a simulated annealing algorithm to optimize the kernel function and regularization function parameters in LS-SVM, and establishes a wireless channel path loss model based on LS-SVM. Comparing the optimized and improved model based on the SA algorithm with the traditional model, the MATLAB simulation experiment results show that the SA+LS-SVM model is significantly superior to the pre-optimized model and the traditional model in prediction accuracy, and can more effectively predict the fading change to the channel Condition.

Prediction models with graph kernel regularization for network data

Traditional regression methods typically consider only covariate information and assume that the observations are mutually independent samples. However, samples usually come from individuals connected by a network in many modern applications. We present a risk minimization formulation for learning from both covariates and network structure in the context of graph kernel regularization. The formulation involves a loss function with a penalty term. This penalty can be used not only to encourage similarity between linked nodes but also lead to improvement over traditional regression models. Furthermore, the penalty can be used with many loss-based predictive methods, such as linear regression with squared loss and logistic regression with log-likelihood loss. Simulations to evaluate the performance of this model in the cases of low dimensions and high dimensions show that our proposed approach outperforms all other benchmarks. We verify this for uniform graph, nonuniform graph, balanced-sample, and unbalanced-sample datasets. The approach was applied to predicting the response values on a ‘follow’ social network of Tencent Weibo users and on two citation networks (Cora and CiteSeer). Each instance verifies that the proposed method combining covariate information and link structure with the graph kernel regularization can improve predictive performance.

A New Methodology to Arrive at Membership Weights for Fuzzy SVM

Support Vector Machine (SVM) is a supervised classification technique that uses the regularization parameter and Kernel function in deciding the best hyperplane to classify the data. SVM is sensitive to outliers, and it makes the model weak. To overcome the issue, the Fuzzy Support Vector Machine (FSVM) introduces fuzzy membership weight into its objective function, which focuses on grouping the fuzzy data points accurately. Knowing the importance of the membership weights in FSVM, we have introduced four new expressions to compute the FSVM membership weights in this study. They are determined from the Fuzzy C-means Algorithm's membership values (FCM). The performances of FSVM with three different kernels are assessed in terms of accuracy. The experiments are conducted for various combinations of FSVM parameters, and the best combinations for each kernel are highlighted. Six benchmark datasets are used to demonstrate the performance of FSVM and the proposed models’ performance are compared with the existing models in recent literature.

Prediction of Carbon Dioxide Solubility in Polymers Based on Adaptive Particle Swarm Optimization and Least Squares Support Vector Machine

AbstractSolubility is a significant physical and chemical property. The solubility of carbon dioxide(CO2) in polymers is an important application of green chemistry. Aimed at the problem of insufficient precision of the existing prediction model, a solubility prediction model based on the adaptive particle swarm optimization algorithm and the least‐squares support vector machine(APSO‐LSSVM) is proposed. Different from the traditional particle swarm algorithm, APSO algorithm improves the problem of easily falling into local optimal solution. The regularization parameters and kernel function tuning parameters of the LSSVM were optimized by APSO and then use this model to predict the solubility of CO2 in eight polymers within a wide range of temperatures and pressures. APSO‐LSSVM model has great prediction ability, with a good correlation between prediction data and experimental data, high prediction accuracy, short calculation time and strong stability. Compared with back propagation artificial neural network(BP ANN), back propagation‐particle swarm optimization algorithm artificial neural network(BP‐PSO ANN) and LSSVM, APSO‐LSSVM has better comprehensive performance. The average absolute relative deviation(AARD), root mean square error(RMSE), determination coefficient(R2) were respectively 0.2130, 0.0120, 0.9853. In addition, the model has good expansibility and can be applied to other fields such as chemistry and medicine.

A new methodology to arrive at membership weights for Fuzzy SVM

Support Vector Machine (SVM) is a supervised classification technique that uses the regularization parameter and Kernel function in deciding the best hyperplane to classify the data. SVM is sensitive to outliers, and it makes the model weak. To overcome the issue, the Fuzzy Support Vector Machine (FSVM) introduces fuzzy membership weight into its objective function, which focuses on grouping the fuzzy data points accurately. Knowing the importance of the membership weights in FSVM, we have introduced four new expressions to compute the FSVM membership weights in this study. They are determined from the Fuzzy C-means Algorithm's membership values (FCM). The performances of FSVM with three different kernels are assessed in terms of accuracy. The experiments are conducted for various combinations of FSVM parameters, and the best combinations for each kernel are highlighted. Six benchmark datasets are used to demonstrate the performance of FSVM and the proposed models’ performance are compared with the existing models in recent literature.

Short-term wind power prediction based on hybrid variational mode decomposition and least squares support vector machine optimized by improved salp swarm algorithm model

To improve the accuracy of wind power prediction, a short-term wind power prediction model based on variational mode decomposition (VMD) and improved salp swarm algorithm (ISSA) optimized least squares support vector machine (LSSVM) is proposed. In the model, the variational modal decomposition is used to decompose the wind power sequence into multiple eigenmode components with limited bandwidth. The improved salp swarm algorithm is employed to tune the regularization parameter and kernel parameter in LSSVM. The proposed wind power prediction strategy using mean one-hour historical wind power data collected from a wind farm located in zhejiang, China. Compared with other prediction models illustrate the better prediction performance of VMD-ISSA-LSSVM.

PaIRKAT: A pathway integrated regression-based kernel association test with applications to metabolomics and COPD phenotypes.

High-throughput data such as metabolomics, genomics, transcriptomics, and proteomics have become familiar data types within the "-omics" family. For this work, we focus on subsets that interact with one another and represent these "pathways" as graphs. Observed pathways often have disjoint components, i.e., nodes or sets of nodes (metabolites, etc.) not connected to any other within the pathway, which notably lessens testing power. In this paper we propose the Pathway Integrated Regression-based Kernel Association Test (PaIRKAT), a new kernel machine regression method for incorporating known pathway information into the semi-parametric kernel regression framework. This work extends previous kernel machine approaches. This paper also contributes an application of a graph kernel regularization method for overcoming disconnected pathways. By incorporating a regularized or "smoothed" graph into a score test, PaIRKAT can provide more powerful tests for associations between biological pathways and phenotypes of interest and will be helpful in identifying novel pathways for targeted clinical research. We evaluate this method through several simulation studies and an application to real metabolomics data from the COPDGene study. Our simulation studies illustrate the robustness of this method to incorrect and incomplete pathway knowledge, and the real data analysis shows meaningful improvements of testing power in pathways. PaIRKAT was developed for application to metabolomic pathway data, but the techniques are easily generalizable to other data sources with a graph-like structure.

Optimized Bayesian adaptive resonance theory mapping model using a rational quadratic kernel and Bayesian quadratic regularization

Optimized Bayesian adaptive resonance theory mapping model using a rational quadratic kernel and Bayesian quadratic regularization

Hilbert–Schmidt Independence Criterion Regularization Kernel Framework on Symmetric Positive Definite Manifolds

Image recognition tasks involve an increasingly high amount of symmetric positive definite (SPD) matrices data. SPD manifolds exhibit nonlinear geometry, and Euclidean machine learning methods cannot be directly applied to SPD manifolds. The kernel trick of SPD manifolds is based on the concept of projecting data onto a reproducing kernel Hilbert space. Unfortunately, existing kernel methods do not consider the connection of SPD matrices and linear projections. Thus, a framework that uses the correlation between SPD matrices and projections to model the kernel map is proposed herein. To realize this, this paper formulates a Hilbert–Schmidt independence criterion (HSIC) regularization framework based on the kernel trick, where HSIC is usually used to express the interconnectedness of two datasets. The proposed framework allows us to extend the existing kernel methods to new HSIC regularization kernel methods. Additionally, this paper proposes an algorithm called HSIC regularized graph discriminant analysis (HRGDA) for SPD manifolds based on the HSIC regularization framework. The proposed HSIC regularization framework and HRGDA are highly accurate and valid based on experimental results on several classification tasks.

P-030. Vitamin D modulates inflammasomes in placental explants of preeclamptic women and decreased apoptosis in HUVEC cultured with monosodium urate

This work discusses about an efficient implementation of the kernel regularization method. In particular, this work focuses on Alternating Direction Method of Multipliers (ADMM) which is one of the convex optimization methods. This work employs two assumptions; (1) the identification input is periodic, and (2) the kernel matrix is tridiagonal. It is shown that ADMM updates can be implemented efficiently under these assumptions. A numerical example is shown to demonstrate the effectiveness of the proposed method.

Deep Convolutional Neural Network Regularization for Alcoholism Detection Using EEG Signals.

Alcoholism is attributed to regular or excessive drinking of alcohol and leads to the disturbance of the neuronal system in the human brain. This results in certain malfunctioning of neurons that can be detected by an electroencephalogram (EEG) using several electrodes on a human skull at appropriate positions. It is of great interest to be able to classify an EEG activity as that of a normal person or an alcoholic person using data from the minimum possible electrodes (or channels). Due to the complex nature of EEG signals, accurate classification of alcoholism using only a small dataset is a challenging task. Artificial neural networks, specifically convolutional neural networks (CNNs), provide efficient and accurate results in various pattern-based classification problems. In this work, we apply CNN on raw EEG data and demonstrate how we achieved 98% average accuracy by optimizing a baseline CNN model and outperforming its results in a range of performance evaluation metrics on the University of California at Irvine Machine Learning (UCI-ML) EEG dataset. This article explains the stepwise improvement of the baseline model using the dropout, batch normalization, and kernel regularization techniques and provides a comparison of the two models that can be beneficial for aspiring practitioners who aim to develop similar classification models in CNN. A performance comparison is also provided with other approaches using the same dataset.

A unified hybrid lattice-Boltzmann method for compressible flows: Bridging between pressure-based and density-based methods

A unified expression for high-speed compressible segregated consistent lattice Boltzmann methods, namely, pressure-based and improved density-based methods, is given. It is theoretically proved that in the absence of forcing terms, these approaches are strictly identical and can be recast in a unique form. An important result is that the difference with classical density-based methods lies in the addition of fourth-order term in the equilibrium function. It is also shown that forcing terms used to balance numerical errors in both original pressure-based and improved density-based methods can be written in a generalized way. A hybrid segregated efficient lattice-Boltzmann for compressible flow based on this unified model, equipped with a recursive regularization kernel, is proposed and successfully assessed on a wide set of test cases with and without shock waves.

An effective health indicator for the Pelton wheel using a Levy flight mutated genetic algorithm

Fluctuations in the head, discharge, and contaminants in the flow can damage parts of the Pelton wheel. An artificial intelligence technique has been investigated for the automatic detection of bucket faults in the Pelton wheel. Features sensitive to defect conditions are extracted from the raw vibration signal and its variational mode decomposition (VMD). The issue of slow convergence speed of the genetic algorithm during optimization is duly addressed by implementing a Levy flight mutated genetic algorithm (LFMGA) while finding the optimal parameters (regularization parameter and kernel function) of a support vector machine (SVM). The efficacy of the proposed LFMGA is tested against different optimization benchmark functions. The results indicate that the proposed algorithm is stable on the basis of the small standard deviation. Using optimized SVM parameters, the SVM model is trained to prepare a classification model with 10-fold cross-validation. After training, the SVM model is tested for fitness evaluation. The overall recognition rate of the SVM model for identification of defects is found to be 98.84% with training time 27.06 s per iteration. A healthy condition is also compared with splitter wear, added mass defect, and missing bucket conditions separately using the VMD–SVM model and shows a recognition rate of 99.17%, 98.33%, and 98.12%, respectively.

A Hybrid Gearbox Fault Diagnosis Method Based on GWO-VMD and DE-KELM

In this paper, a vibration signal-based hybrid diagnostic method, including vibration signal adaptive decomposition, vibration signal reconstruction, fault feature extraction, and gearbox fault classification, is proposed to realize fault diagnosis of general gearboxes. The main contribution of the proposed method is the combining of signal processing, machine learning, and optimization techniques to effectively eliminate noise contained in vibration signals and to achieve high diagnostic accuracy. Firstly, in the study of vibration signal preprocessing and fault feature extraction, to reduce the impact of noise and mode mixing problems on the accuracy of fault classification, Variational Mode Decomposition (VMD) was adopted to realize adaptive signal decomposition and Wolf Grey Optimizer (GWO) was applied to optimize parameters of VMD. The correlation coefficient was subsequently used to select highly correlated Intrinsic Mode Functions (IMFs) to reconstruct the vibration signals. With these re-constructed signals, fault features were extracted by calculating their time domain parameters, energies, and permutation entropies. Secondly, in the study of fault classification, Kernel Extreme Learning Machine (KELM) was adopted and Differential Evolutionary (DE) was applied to search its regularization coefficient and kernel parameter to further improve classification accuracy. Finally, gearbox vibration signals in healthy and faulty conditions were obtained and contrast experiences were conducted to validate the effectiveness of the proposed hybrid fault diagnosis method.

Optimization of regularization coefficient and kernel parameters of KELM in face recognition using genetic algorithm

Extreme Learning machine (ELM), a linear system model introduced originally for feedforward neural networks having single-hidden layer, which is non-iteratively tuned. In ELM feature mapping is achieved explicitly with the help of activation functions. ELM was further prospected with kernel functions for performance improvement by exploiting implicit feature mapping. The classification accuracy of kernel ELM relies on the kernel and structural parameters, which are experimentally tuned. It is very challenging and computationally hard to assess the optimal choice of these parameters from a specific domain of values using Brute force method. Therefore an optimal algorithm is required to find the finest combination of these parameters for enhanced performance. In this paper optimized kernel ELM is unveiled, in which genetic algorithm is utilized to evaluate the optimal solution of regularization coefficient and kernel parameters. Experiments on face image databases denote that the developed algorithm is more accurate and efficacious than state of art existing algorithms in literature.

General-purpose kernel regularization of boundary integral equations via density interpolation

This paper presents a general high-order kernel regularization technique applicable to all four integral operators of Calderón calculus associated with linear elliptic PDEs in two and three spatial dimensions. Like previous density interpolation methods, the proposed technique relies on interpolating the density function around the kernel singularity in terms of solutions of the underlying homogeneous PDE, so as to recast singular and nearly singular integrals in terms of bounded (or more regular) integrands. We present here a simple interpolation strategy which, unlike previous approaches, does not entail explicit computation of high-order derivatives of the density function along the surface. Furthermore, the proposed approach is kernel- and dimension-independent in the sense that the sought density interpolant is constructed as a linear combination of point-source fields, given by the same Green’s function used in the integral equation formulation, thus making the procedure applicable, in principle, to any PDE with known Green’s function. For the sake of definiteness, we focus here on Nystr‘̀om methods for the (scalar) Laplace and Helmholtz equations and the (vector) elastostatic and time-harmonic elastodynamic equations. The method’s accuracy, flexibility, efficiency, and compatibility with fast solvers are demonstrated by means of a variety of large-scale three-dimensional numerical examples.

A Novel Matrix Completion Model Based on the Multi-Layer Perceptron Integrating Kernel Regularization

Matrix completion has been widely used in image recovery and recommendation. The conventional matrix completion models based on multi-layer perceptron (MLP) only has local constraints on the observation data so that the completed matrix contains a lot of noise. Therefore, this paper proposes a novel matrix completion model based on the MLP integrating kernel regularization (MCKR). The proposed model uses the MLP to extract feature, which can automatically extract interaction feature of missing data and observational data. Moreover, through a non-linear mapping of low-dimensional latent subspace and schatten p-norm, a novel kernel regularization is derived to provide a global constraint for reducing the noise of the completed matrix. Finally, the validity and practicability of the proposed model is verified on three publicly available datasets. Compared with baselines including matrix factorization (MF), Logistic matrix factorization (LMF), expertise matrix factorization (EMF) and multi-layer perceptron (MLP), the proposed model achieves state-of-the-art result.

Visible-to-Thermal Transfer Learning for Facial Landmark Detection

There has been increasing interest in face recognition in the thermal infrared spectrum. A critical step in this process is face landmark detection. However, landmark detection in the thermal spectrum presents a unique set of challenges compared to in the visible spectrum: inherently lower spatial resolution due to longer wavelength, differences in phenomenology, and limited availability of labeled thermal face imagery for algorithm development and training. Thermal infrared imaging does have the advantage of being able to passively acquire facial heat signatures without the need for active or ambient illumination in low light and nighttime environments. In such scenarios, thermal imaging must operate by itself without corresponding/paired visible imagery. Mindful of this constraint, we propose visible-to-thermal parameter transfer learning using a coupled convolutional network architecture as a means to leverage visible face data when training a model for thermal-only face landmark detection. This differentiates our approach from models trained either solely on thermal images or models which require a fusion of visible and thermal images at test time. In this work, we implement and analyze four types of parameter transfer learning methods in the context of thermal face landmark detection: Siamese (shared) layers, Linear Layer Regularization (LLR), Linear Kernel Regularization (LKR), and Residual Parameter Transformations (RPT). These transfer learning approaches are compared against a baseline version of the network and an Active Appearance Model (AAM), both of which are trained only on thermal data. We achieve a 6.5% - 9.5% improvement on the DEVCOM ARL Multi-modal Thermal Face Dataset and a 4% improvement on the RWTH Aachen University Thermal Face Dataset over the baseline model. We show that LLR, LKR, and RPT all result in improved thermal face landmark detection performance compared to the baseline and AAM, demonstrating that transfer learning leveraging visible spectrum data improves thermal face landmarking.

Efficient Implementation of Kernel Regularization based on ADMM

Abstract This work discusses about an efficient implementation of the kernel regularization method. In particular, this work focuses on Alternating Direction Method of Multipliers (ADMM) which is one of the convex optimization methods. This work employs two assumptions; (1) the identification input is periodic, and (2) the kernel matrix is tridiagonal. It is shown that ADMM updates can be implemented efficiently under these assumptions. A numerical example is shown to demonstrate the effectiveness of the proposed method.