Kernel-based Machine Learning Algorithms Research Articles

Do energy, inflation, and financial development stimulate economic welfare in India? Empirical insights from novel dynamic ARDL and KRLS simulations

The present study focuses on investigating the effect of renewable and non-renewable energy consumption (REC and NREC), financial development (FD), and inflation (INF) on economic welfare (EW) in India during 1990–2019. In this research, we employ a novel dynamic autoregressive distributed lag model and a kernel-based machine learning algorithm and regularise least squares to detect causal linkages among the variables. To measure the economic welfare index, this research uses the PCA analysis with GDP, human capital, environmental damage, number of employed persons, life expectancy at birth, and real consumption in households and by the government. According to the findings, the NREC and FD have been favourably contributing to EW. Conversely, the EW is adversely affected by REC. However, INF also has an adverse impact on EW, although it may be insignificant in the long run and significant in the short run.

Fast, accurate, and interpretable decoding of electrocorticographic signals using dynamic mode decomposition

Dynamic mode (DM) decomposition decomposes spatiotemporal signals into basic oscillatory components (DMs). DMs can improve the accuracy of neural decoding when used with the nonlinear Grassmann kernel, compared to conventional power features. However, such kernel-based machine learning algorithms have three limitations: large computational time preventing real-time application, incompatibility with non-kernel algorithms, and low interpretability. Here, we propose a mapping function corresponding to the Grassmann kernel that explicitly transforms DMs into spatial DM (sDM) features, which can be used in any machine learning algorithm. Using electrocorticographic signals recorded during various movement and visual perception tasks, the sDM features were shown to improve the decoding accuracy and computational time compared to conventional methods. Furthermore, the components of the sDM features informative for decoding showed similar characteristics to the high-γ power of the signals, but with higher trial-to-trial reproducibility. The proposed sDM features enable fast, accurate, and interpretable neural decoding.

Biophysical Variable Retrieval of Silage Maize with Gaussian Process Regression and Hyperparameter Optimization Algorithms

Quantification of vegetation biophysical variables such as leaf area index (LAI), fractional vegetation cover (fCover), and biomass are among the key factors across hydrological, agricultural, and irrigation management studies. The present study proposes a kernel-based machine learning algorithm capable of performing adaptive and nonlinear data fitting so as to generate a suitable, accurate, and robust algorithm for spatio-temporal estimation of the three mentioned variables using Sentinel-2 images. To this aim, Gaussian process regression (GPR)–particle swarm optimization (PSO), GPR–genetic algorithm (GA), GPR–tabu search (TS), and GPR–simulated annealing (SA) hyperparameter-optimized algorithms were developed and compared against kernel-based machine learning regression algorithms and artificial neural network (ANN) and random forest (RF) algorithms. The accuracy of the proposed algorithms was assessed using digital hemispherical photography (DHP) data and destructive measurements performed during the growing season of silage maize in agricultural fields of Ghale-Nou, southern Tehran, Iran, in the summer of 2019. The results on biophysical variables against validation data showed that the developed GPR-PSO algorithm outperformed other algorithms under study in terms of robustness and accuracy (0.917, 0.931, 0.882 using R2 and 0.627, 0.078, and 1.99 using RMSE in LAI, fCover, and biomass of Sentinel-2 20 m, respectively). GPR-PSO also possesses the unique ability to generate pixel-based uncertainty maps (confidence level) for prediction purposes (i.e., estimated uncertainty level <0.7 in LAI, fCover, and biomass, for 96%, 98%, and 71% of the total study area, respectively). Altogether, GPR-PSO appears to be the most suitable option for mapping biophysical variables at the local scale using Sentinel-2 images.

Brain-age prediction: A systematic comparison of machine learning workflows

The difference between age predicted using anatomical brain scans and chronological age, i.e., the brain-age delta, provides a proxy for atypical aging. Various data representations and machine learning (ML) algorithms have been used for brain-age estimation. However, how these choices compare on performance criteria important for real-world applications, such as; (1) within-dataset accuracy, (2) cross-dataset generalization, (3) test-retest reliability, and (4) longitudinal consistency, remains uncharacterized. We evaluated 128 workflows consisting of 16 feature representations derived from gray matter (GM) images and eight ML algorithms with diverse inductive biases. Using four large neuroimaging databases covering the adult lifespan (total N = 2953, 18–88 years), we followed a systematic model selection procedure by sequentially applying stringent criteria. The 128 workflows showed a within-dataset mean absolute error (MAE) between 4.73–8.38 years, from which 32 broadly sampled workflows showed a cross-dataset MAE between 5.23–8.98 years. The test-retest reliability and longitudinal consistency of the top 10 workflows were comparable. The choice of feature representation and the ML algorithm both affected the performance. Specifically, voxel-wise feature spaces (smoothed and resampled), with and without principal components analysis, with non-linear and kernel-based ML algorithms performed well. Strikingly, the correlation of brain-age delta with behavioral measures disagreed between within-dataset and cross-dataset predictions. Application of the best-performing workflow on the ADNI sample showed a significantly higher brain-age delta in Alzheimer's and mild cognitive impairment patients compared to healthy controls. However, in the presence of age bias, the delta estimates in the patients varied depending on the sample used for bias correction. Taken together, brain-age shows promise, but further evaluation and improvements are needed for its real-world application.

Fast and interpretable genomic data analysis using multiple approximate kernel learning.

MotivationDataset sizes in computational biology have been increased drastically with the help of improved data collection tools and increasing size of patient cohorts. Previous kernel-based machine learning algorithms proposed for increased interpretability started to fail with large sample sizes, owing to their lack of scalability. To overcome this problem, we proposed a fast and efficient multiple kernel learning (MKL) algorithm to be particularly used with large-scale data that integrates kernel approximation and group Lasso formulations into a conjoint model. Our method extracts significant and meaningful information from the genomic data while conjointly learning a model for out-of-sample prediction. It is scalable with increasing sample size by approximating instead of calculating distinct kernel matrices.ResultsTo test our computational framework, namely, Multiple Approximate Kernel Learning (MAKL), we demonstrated our experiments on three cancer datasets and showed that MAKL is capable to outperform the baseline algorithm while using only a small fraction of the input features. We also reported selection frequencies of approximated kernel matrices associated with feature subsets (i.e. gene sets/pathways), which helps to see their relevance for the given classification task. Our fast and interpretable MKL algorithm producing sparse solutions is promising for computational biology applications considering its scalability and highly correlated structure of genomic datasets, and it can be used to discover new biomarkers and new therapeutic guidelines.Availability and implementationMAKL is available at https://github.com/begumbektas/makl together with the scripts that replicate the reported experiments. MAKL is also available as an R package at https://cran.r-project.org/web/packages/MAKL.Supplementary information Supplementary data are available at Bioinformatics online.

Machine-Learning-Based Semiparametric Time Series Conditional Variance: Estimation and Forecasting

This paper proposes a new combined semiparametric estimator of the conditional variance that takes the product of a parametric estimator and a nonparametric estimator based on machine learning. A popular kernel-based machine learning algorithm, known as the kernel-regularized least squares estimator, is used to estimate the nonparametric component. We discuss how to estimate the semiparametric estimator using real data and how to use this estimator to make forecasts for the conditional variance. Simulations are conducted to show the dominance of the proposed estimator in terms of mean squared error. An empirical application using S&P 500 daily returns is analyzed, and the semiparametric estimator effectively forecasts future volatility.

ESCAPED: Efficient Secure and Private Dot Product Framework for Kernel-based Machine Learning Algorithms with Applications in Healthcare

Training sophisticated machine learning models usually requires many training samples. Especially in healthcare settings these samples can be very expensive, meaning that one institution alone usually does not have enough. Merging privacy-sensitive data from different sources is usually restricted by data security and data protection measures. This can lead to approaches that reduce data quality by putting noise onto the variables (e.g., in epsilon-differential privacy) or omitting certain values (e.g., for k-anonymity). Other measures based on cryptographic methods can lead to very time-consuming computations, which is especially problematic for larger multi-omics data. We address this problem by introducing ESCAPED, which stands for Efficient SeCure And PrivatE Dot product framework. ESCAPED enables the computation of the dot product of vectors from multiple sources on a third-party, which later trains kernel-based machine learning algorithms, while neither sacrificing privacy nor adding noise. We have evaluated our framework on drug resistance prediction for HIV-infected people and multi-omics dimensionality reduction and clustering problems in precision medicine. In terms of execution time, our framework significantly outperforms the best-fitting existing approaches without sacrificing the performance of the algorithm. Even though we only present the benefit for kernel-based algorithms, our framework can open up new research opportunities for further machine learning models that require the dot product of vectors from multiple sources.

Dimensionality Reduction of Complex Metastable Systems via Kernel Embeddings of Transition Manifolds

We present a novel kernel-based machine learning algorithm for identifying the low-dimensional geometry of the effective dynamics of high-dimensional multiscale stochastic systems. Recently, the authors developed a mathematical framework for the computation of optimal reaction coordinates of such systems that is based on learning a parameterization of a low-dimensional transition manifold in a certain function space. In this article, we enhance this approach by embedding and learning this transition manifold in a reproducing kernel Hilbert space, exploiting the favorable properties of kernel embeddings. Under mild assumptions on the kernel, the manifold structure is shown to be preserved under the embedding, and distortion bounds can be derived. This leads to a more robust and more efficient algorithm compared to the previous parameterization approaches.

Prediction of arch dam deformation via correlated multi-target stacking

Majority of the existing dam deformation monitoring models focus on the prediction of individual displacement, and ignore the spatial correlation of data. In this study, we propose a method dealing with multi-target prediction called the Maximum Correlated Stacking of Single-Target. The proposed method can provide reliable predictions of multi-target simultaneously, while fully exploiting the internal relationships between target variables via the strategy of targets stacking. Moreover, it can be coupled with different existing baseline models for the prediction and anomaly detection of arch dam deformation.Jinping–I arch dam is taken as a case study, where the monitoring displacement of 23 different points are analyzed and modeled simultaneously. Three kernel-based machine learning algorithms (i.e., support vector machine, relevance vector machine, and kernel extreme learning machine) and the partial least squares regression are adopted as baseline models for multi-target regression methods. Compared with the single-target regression and two state-of-the-art multi-target regression methods, the simulated results reveal the higher accuracy of the proposed method. Furthermore, model performance is validated in terms of anomaly detection capability, where two progressive anomalous scenarios (i.e., anomalies of single or multiple points) are investigated. The proposed method can be adapted for the health monitoring of other infrastructures in which multiple responses (e.g., displacement, temperature, or stress) need to be predicted simultaneously.

Scalable nanolaminated SERS multiwell cell culture assay

This paper presents a new cell culture platform enabling label-free surface-enhanced Raman spectroscopy (SERS) analysis of biological samples. The platform integrates a multilayered metal-insulator-metal nanolaminated SERS substrate and polydimethylsiloxane (PDMS) multiwells for the simultaneous analysis of cultured cells. Multiple cell lines, including breast normal and cancer cells and prostate cancer cells, were used to validate the applicability of this unique platform. The cell lines were cultured in different wells. The Raman spectra of over 100 cells from each cell line were collected and analyzed after 12 h of introducing the cells to the assay. The unique Raman spectra of each cell line yielded biomarkers for identifying cancerous and normal cells. A kernel-based machine learning algorithm was used to extract the high-dimensional variables from the Raman spectra. Specifically, the nonnegative garrote on a kernel machine classifier is a hybrid approach with a mixed nonparametric model that considers the nonlinear relationships between the higher-dimension variables. The breast cancer cell lines and normal breast epithelial cells were distinguished with an accuracy close to 90%. The prediction rate between breast cancer cells and prostate cancer cells reached 94%. Four blind test groups were used to evaluate the prediction power of the SERS spectra. The peak intensities at the selected Raman shifts of the testing groups were selected and compared with the training groups used in the machine learning algorithm. The blind testing groups were correctly predicted 100% of the time, demonstrating the applicability of the multiwell SERS array for analyzing cell populations for cancer research.

Quantitative robustness of localized support vector machines

The huge amount of available data nowadays is a challenge for kernel-based machine learning algorithms like SVMs with respect to runtime and storage capacities. Local approaches might help to relieve these issues and to improve statistical accuracy. It has already been shown that these local approaches are consistent and robust in a basic sense. This article refines the analysis of robustness properties towards the so-called influence function which expresses the differentiability of the learning method: We show that there is a differentiable dependency of our locally learned predictor on the underlying distribution. The assumptions of the proven theorems can be verified without knowing anything about this distribution. This makes the results interesting also from an applied point of view.

Machine Learning-Based Handovers for Sub-6 GHz and mmWave Integrated Vehicular Networks

The integration of sub-6 GHz and millimeter wave (mmWave) bands has a great potential to enable both reliable coverage and high data rate in future vehicular networks. Nevertheless, during mmWave vehicle-to-infrastructure (V2I) handovers, the coverage blindness of directional beams makes it a significant challenge to discover target mmWave remote radio units (mmW-RRUs) whose active beams may radiate somewhere that the handover vehicles are not in. Besides, fast and soft handovers are also urgently needed in vehicular networks. Based on these observations, to solve the target discovery problem, we utilize channel state information (CSI) of sub-6 GHz bands and Kernel-based machine learning (ML) algorithms to predict vehicles’ positions and then use them to pre-activate target mmW-RRUs. Considering that the regular movement of vehicles on almost linearly paved roads with finite corner turns will generate some regularity in handovers, to accelerate handovers, we propose to use historical handover data and $K$ -nearest neighbor (KNN) ML algorithms to predict handover decisions without involving time-consuming target selection and beam training processes. To achieve soft handovers, we propose to employ vehicle-to-vehicle (V2V) connections to forward data for V2I links. The theoretical and simulation results are provided to validate the feasibility of the proposed schemes.

Discrete and continuous emotion recognition using sequence kernels

The field of automatic speech emotion recognition is a highly active and multi-diverse research area. The current state-of-the-art approach in machine analysis of human emotion has focused on recognition of discrete emotional states, such as the six basic emotion categories. However, emotion is deemed complex and is characterised in terms of latent dimensions. Accordingly, this paper aims at recognising discrete and continuous emotional states by adapting the emotional recognition system to the advanced kernel-based machine learning algorithms from the field of speaker recognition, we argue that it is more efficient in terms of recognition performance. The focus in this paper is to build a range of sequence kernel to recognise discrete and continuous emotions from the well-established real-life speech dataset (IEMOCAP) and the acted Berlin emotional speech dataset (Emo-DB).

Bayesian Correlation Analysis for Sequence Count Data.

Evaluating the similarity of different measured variables is a fundamental task of statistics, and a key part of many bioinformatics algorithms. Here we propose a Bayesian scheme for estimating the correlation between different entities’ measurements based on high-throughput sequencing data. These entities could be different genes or miRNAs whose expression is measured by RNA-seq, different transcription factors or histone marks whose expression is measured by ChIP-seq, or even combinations of different types of entities. Our Bayesian formulation accounts for both measured signal levels and uncertainty in those levels, due to varying sequencing depth in different experiments and to varying absolute levels of individual entities, both of which affect the precision of the measurements. In comparison with a traditional Pearson correlation analysis, we show that our Bayesian correlation analysis retains high correlations when measurement confidence is high, but suppresses correlations when measurement confidence is low—especially for entities with low signal levels. In addition, we consider the influence of priors on the Bayesian correlation estimate. Perhaps surprisingly, we show that naive, uniform priors on entities’ signal levels can lead to highly biased correlation estimates, particularly when different experiments have widely varying sequencing depths. However, we propose two alternative priors that provably mitigate this problem. We also prove that, like traditional Pearson correlation, our Bayesian correlation calculation constitutes a kernel in the machine learning sense, and thus can be used as a similarity measure in any kernel-based machine learning algorithm. We demonstrate our approach on two RNA-seq datasets and one miRNA-seq dataset.

Kernel subspace pursuit for sparse regression

Recently, results from sparse approximation theory have been considered as a means to improve the generalization performance of kernel-based machine learning algorithms. In this paper, we present Kernel Subspace Pursuit (KSP), a new method for sparse non-linear regression. KSP is a low-complexity method that iteratively approximates target functions in the least-squares sense as a linear combination of a limited number of elements selected from a kernel-based dictionary. Unlike other kernel methods, by virtue of KSP’s algorithmic design, the number of KSP iterations needed to reach the final solution does not depend on the number of basis functions used nor that of elements in the dictionary. We experimentally show that, in many scenarios involving learning synthetic and real data, KSP is less complex computationally and outperforms other kernel methods that solve the same problem, namely, Kernel Matching Pursuit and Kernel Basis Pursuit.

Local online kernel ridge regression for forecasting of urban travel times

Accurate and reliable forecasting of traffic variables is one of the primary functions of Intelligent Transportation Systems. Reliable systems that are able to forecast traffic conditions accurately, multiple time steps into the future, are required for advanced traveller information systems. However, traffic forecasting is a difficult task because of the nonlinear and nonstationary properties of traffic series. Traditional linear models are incapable of modelling such properties, and typically perform poorly, particularly when conditions differ from the norm. Machine learning approaches such as artificial neural networks, nonparametric regression and kernel methods (KMs) have often been shown to outperform linear models in the literature. A bottleneck of the latter approach is that the information pertaining to all previous traffic states must be contained within the kernel, but the computational complexity of KMs usually scales cubically with the number of data points in the kernel. In this paper, a novel kernel-based machine learning (ML) algorithm is developed, namely the local online kernel ridge regression (LOKRR) model. Exploiting the observation that traffic data exhibits strong cyclic patterns characterised by rush hour traffic, LOKRR makes use of local kernels with varying parameters that are defined around each time point. This approach has 3 advantages over the standard single kernel approach: (1) It allows parameters to vary by time of day, capturing the time varying distribution of traffic data; (2) It allows smaller kernels to be defined that contain only the relevant traffic patterns, and; (3) It is online, allowing new traffic data to be incorporated as it arrives. The model is applied to the forecasting of travel times on London’s road network, and is found to outperform three benchmark models in forecasting up to 1h ahead.

Gaussian process models for reference ET estimation from alternative meteorological data sources

Summary Accurate estimates of daily crop evapotranspiration (ET) are needed for efficient irrigation management, especially in arid and semi-arid regions where crop water demand exceeds rainfall. Daily grass or alfalfa reference ET values and crop coefficients are widely used to estimate crop water demand. Inaccurate reference ET estimates can hence have a tremendous impact on irrigation costs and the demands on U.S. freshwater resources, particularly within the Ogallala aquifer region. ET networks calculate reference ET using local meteorological data. With gaps in spatial coverage of existing networks and the agriculture-based Texas High Plains ET (TXHPET) network in jeopardy due to lack of funding, there is an immediate need for alternative sources capable of filling data gaps without high maintenance and field-based support costs. Non-agricultural weather stations located throughout the Texas High Plains are providing publicly accessible meteorological data. However, there are concerns that the data may not be suitable for estimating reference ET due to factors such as weather station siting, fetch requirements, data formats, parameters recorded, and quality control issues. The goal of the research reported in this paper is to assess the use of alternative data sources for reference ET computation. Towards this objective, we trained Gaussian process models, an instance of kernel-based machine learning algorithms, on data collected from weather stations to estimate reference ET values and augment the TXHPET database. Results show that Gaussian process models provide much greater accuracy than baseline least square regression models.

Monthly streamflow forecasting using Gaussian Process Regression

Streamflow forecasting plays a critical role in nearly all aspects of water resources planning and management. In this work, Gaussian Process Regression (GPR), an effective kernel-based machine learning algorithm, is applied to probabilistic streamflow forecasting. GPR is built on Gaussian process, which is a stochastic process that generalizes multivariate Gaussian distribution to infinite-dimensional space such that distributions over function values can be defined. The GPR algorithm provides a tractable and flexible hierarchical Bayesian framework for inferring the posterior distribution of streamflows. The prediction skill of the algorithm is tested for one-month-ahead prediction using the MOPEX database, which includes long-term hydrometeorological time series collected from 438 basins across the U.S. from 1948 to 2003. Comparisons with linear regression and artificial neural network models indicate that GPR outperforms both regression methods in most cases. The GPR prediction of MOPEX basins is further examined using the Budyko framework, which helps to reveal the close relationships among water-energy partitions, hydrologic similarity, and predictability. Flow regime modification and the resulting loss of predictability have been a major concern in recent years because of climate change and anthropogenic activities. The persistence of streamflow predictability is thus examined by extending the original MOPEX data records to 2012. Results indicate relatively strong persistence of streamflow predictability in the extended period, although the low-predictability basins tend to show more variations. Because many low-predictability basins are located in regions experiencing fast growth of human activities, the significance of sustainable development and water resources management can be even greater for those regions.

The continuous molecular fields approach to building 3D-QSAR models

The continuous molecular fields (CMF) approach is based on the application of continuous functions for the description of molecular fields instead of finite sets of molecular descriptors (such as interaction energies computed at grid nodes) commonly used for this purpose. These functions can be encapsulated into kernels and combined with kernel-based machine learning algorithms to provide a variety of novel methods for building classification and regression structure-activity models, visualizing chemical datasets and conducting virtual screening. In this article, the CMF approach is applied to building 3D-QSAR models for 8 datasets through the use of five types of molecular fields (the electrostatic, steric, hydrophobic, hydrogen-bond acceptor and donor ones), the linear convolution molecular kernel with the contribution of each atom approximated with a single isotropic Gaussian function, and the kernel ridge regression data analysis technique. It is shown that the CMF approach even in this simplest form provides either comparable or enhanced predictive performance in comparison with state-of-the-art 3D-QSAR methods.

Probabilistic Kernel Approach to Online Monitoring of Nuclear Power Plants

Nuclear power plants are complex engineering systems comprised of many interacting and interdependent mechanical components whose failure might lead to degraded plant performance or unplanned shutdown with loss of power generation and negative economic impact. As a result, continuous component surveillance and accurate prediction of their failing points is necessary for their on-time replacement. In this paper, a probabilistic kernel approach for intelligent online monitoring of mechanical components is presented. Specifically, the probabilistic kernel notion of Gaussian processes (GPs) is applied to the distribution prediction of a component's degradation trend. The proposed method exploits the learning ability of a GP and updates its prediction using a feedback mechanism. The methodology is tested on actual turbine blade degradation data for a variety of topologies (i.e., kernels). The GP estimations are compared to those obtained with a nonprobabilistic, kernel-based machine learning algorithm, the support vector regression (SVR). The comparison outcome clearly demonstrates that GP prediction accuracy outperforms SVR in the majority of the cases while providing a predictive distribution instead of point estimates as SVR does.