Kernel Selection Research Articles

Towards a model-independent reconstruction approach for late-time Hubble data

Gaussian processes offers a convenient way to perform nonparametric reconstructions of observational data assuming only a kernel which describes the covariance between neighbouring points in a data set. We approach the ambiguity in the choice of kernel in Gaussian processes with two methods — (a) approximate Bayesian computation with sequential Monte Carlo sampling and (b) genetic algorithm — and use the overall resulting method to reconstruct the cosmic chronometers and supernovae type Ia data sets.The results have shown that the Matérn( ν = 5/2 ) kernel emerges on top of the two-hyperparameter family of kernels for both cosmological data sets. On the other hand, we use the genetic algorithm in order to select a most naturally-fit kernel among a competitive pool made up of a ten-hyperparameters class of kernels. Imposing a Bayesian information criterion-inspired measure of the fitness, the results have shown that a hybrid of the Radial Basis Function and the Matérn( ν = 5/2 ) kernel best represented both data sets. The kernel selection problem is not totally closed and may benefit from further analysis using other strategies to resolve an optimal kernel for a particular data set.

Deep regularization and direct training of the inner layers of Neural Networks with Kernel Flows

We introduce a new regularization method for Artificial Neural Networks (ANNs) based on the Kernel Flow (KF) algorithm. The algorithm was introduced in Owhadi and Yoo (2019) as a method for kernel selection in regression/kriging based on the minimization of the loss of accuracy incurred by halving the number of interpolation points in random batches of the dataset. Writing fθ(x)=(fθn(n)∘fθn−1(n−1)∘⋯∘fθ1(1))(x) for the functional representation of compositional structure of the ANN (where θi are the weights and biases of the layer i), the inner layers outputs h(i)(x)=(fθi(i)∘fθi−1(i−1)∘…∘fθ1(1))(x) define a hierarchy of feature maps and a hierarchy of kernels k(i)(x,x′)=exp(−γi‖h(i)(x)−h(i)(x′)‖22). When combined with a batch of the dataset, these kernels produce KF losses e2(i) (defined as the L2 regression error incurred by using a random half of the batch to predict the other half) depending on the parameters of the inner layers θ1,…,θi (and γi). The proposed method simply consists of aggregating (as a weighted sum) a subset of these KF losses with a classical output loss (e.g., cross-entropy). We test the proposed method on Convolutional Neural Networks (CNNs) and Wide Residual Networks (WRNs) without alteration of their structure nor their output classifier and report reduced test errors, decreased generalization gaps, and increased robustness to distribution shift without a significant increase in computational complexity relative to standard CNN and WRN training (with Drop Out and Batch Normalization). We suspect that these results might be explained by the fact that while conventional training only employs a linear functional (a generalized moment) of the empirical distribution defined by the dataset and can be prone to trapping in the Neural Tangent Kernel regime (under over-parameterizations), the proposed loss function (defined as a nonlinear functional of the empirical distribution) effectively trains the underlying kernel defined by the CNN beyond regressing the data with that kernel.

Short-term volatility forecasting with kernel support vector regression and Markov switching multifractal model

In volatility forecasting literature, Markov switching multifractal (MSM) models are well known for capturing many important stylized facts such as long memory and fat tails. MSM delivers stronger performance both in- and out-of-sample than GARCH-type models in long-term forecasts. However, the literature shows that MSM forecasts only slightly improve on GARCH(1,1) at short-term intervals. This indicates that there may exist certain patterns to be discovered in the innovation part . To enhance MSM's prediction accuracy at the short-term level with higher frequency data, a hybrid model of the MSM model and support vector regression (SVR) is proposed, in which a particle swarm optimization (PSO) algorithm is applied to optimize hyperparameters of the support vector regression in the scope of constraint permission. The method is referred to as MSM-PSO-SVR. Further, we introduce the Fourier kernel MSM-PSO-SVR and evaluate the performance of various MSM-PSO-SVR models in terms of mean absolute error (MAE) and the mean squared error (MSE) with one-minute data of the exchange traded fund (ETF) SPDR S&P 500 Trust ETF (ticker symbol: SPY). The experimental results show that the proposed approach outperforms the other competing peer models and in particular, the selection of SVR kernel might yield significant boosts in forecasting ability. Results of Hansen's Superior Predictive Ability test further validate the conclusion.

Machine learning techniques for liquid level estimation using FBG temperature sensor array

This paper proposes the use of the fiber Bragg grating (FBG) temperature sensors array to estimate the fluid level. The tank is 100 cm in height and 30 cm in width, with 9 FBG sensors distributed along with the tank height. This work proposes level estimation in two steps: level detection (classification) and level estimation (regression). The level detection consists of finding under which FBG the level is. We dichotomize the classes in water and not water: air. For the detection, we use the following Machine Learning (ML) algorithms: Logistic Regression (LogR), Decision Tree (DT), and Support Vector Machine (SVM). We chose the algorithms based on their usability in literature and theoretical consolidation. The algorithm with the best results among the ones tested is DT, resulting in an average accuracy of 89.54%. For the level estimation, we use the wavelength shift in conjunction with the classification obtained via DT. The level estimation consists of estimating in cm the location of the level. The algorithms chosen are Weighted Linear Regression (WLR), Support Vector Regression (SVR), SVR with kernel selection minimize cost (SVRmin). Each of the applied algorithms presented better results in different level intervals. Thus, we propose the Mixed Model (MM), which selects the lowest Root Mean Square Error (RMSE) among the tested regression algorithms at each level and associates it with it. The MM has a RMSE of 3.56 cm, which is approximately four times smaller than when using WLR. The SVM and SVMmin have RMSE of 6.28 cm and 6.14 cm, respectively.

Deterministic Multi-kernel based extreme learning machine for pattern classification

The Extreme learning machine (ELM) designed by Huang et al. is proved to be a fast and good classifier over a decade, but existing ELM is non-deterministic in nature as well as kernel dependent and needs attention to optimize the selection of kernels. In ELM feature space is obtained with the help of single kernel function. The choice of kernel depends on perceptiveness of classification problem. So a generalized framework with deterministic nature along with optimized kernel is ought to be designed that can be applied to large domain of real world heterogeneous pattern classification problems. This paper presents a deterministic extreme learning machine for neural network with feedforward architecture which is formulated with multiple kernel learning. We further enhance this approach by incorporating Gray level co-occurrence matrix (GLCM) for multi-modal feature extraction. Two formulation of kernel extreme learning machine are introduced, with target kernel function as a linear combination of different base kernels. The first one is based on deterministic multiple kernel learning while the second one uses deterministic multiple kernel learning along with GLCM for extracting the invariant feature vectors. The performance of proposed algorithms are analyzed on pattern recognition problem for face recognition by changing the number of training set, types of kernel used and coefficients used for combining base kernels. The superior recognition rate is achieved for prominent multi-class face databases, when compared with contemporary methods that prove the efficacy of proposed algorithms.

Accurate Retinal Vessel Segmentation in Color Fundus Images via Fully Attention-Based Networks.

Automatic retinal vessel segmentation is important for the diagnosis and prevention of ophthalmic diseases. The existing deep learning retinal vessel segmentation models always treat each pixel equally. However, the multi-scale vessel structure is a vital factor affecting the segmentation results, especially in thin vessels. To address this crucial gap, we propose a novel Fully Attention-based Network (FANet) based on attention mechanisms to adaptively learn rich feature representation and aggregate the multi-scale information. Specifically, the framework consists of the image pre-processing procedure and the semantic segmentation networks. Green channel extraction (GE) and contrast limited adaptive histogram equalization (CLAHE) are employed as pre-processing to enhance the texture and contrast of retinal blood images. Besides, the network combines two types of attention modules with the U-Net. We propose a lightweight dual-direction attention block to model global dependencies and reduce intra-class inconsistencies, in which the weights of feature maps are updated based on the semantic correlation between pixels. The dual-direction attention block utilizes horizontal and vertical pooling operations to produce the attention map. In this way, the network aggregates global contextual information from semantic-closer regions or a series of pixels belonging to the same object category. Meanwhile, we adopt the selective kernel (SK) unit to replace the standard convolution for obtaining multi-scale features of different receptive field sizes generated by soft attention. Furthermore, we demonstrate that the proposed model can effectively identify irregular, noisy, and multi-scale retinal vessels. The abundant experiments on DRIVE, STARE, and CHASE_DB1 datasets show that our method achieves state-of-the-art performance.

Learning Compositional Sparse Gaussian Processes with a Shrinkage Prior

Choosing a proper set of kernel functions is an important problem in learning Gaussian Process (GP) models since each kernel structure has different model complexity and data fitness. Recently, automatic kernel composition methods provide not only accurate prediction but also attractive interpretability through search-based methods. However, existing methods suffer from slow kernel composition learning. To tackle large-scaled data, we propose a new sparse approximate posterior for GPs, MultiSVGP, constructed from groups of inducing points associated with individual additive kernels in compositional kernels. We demonstrate that this approximation provides a better fit to learn compositional kernels given empirical observations. We also provide theoretically justification on error bound when compared to the traditional sparse GP. In contrast to the search-based approach, we present a novel probabilistic algorithm to learn a kernel composition by handling the sparsity in the kernel selection with Horseshoe prior. We demonstrate that our model can capture characteristics of time series with significant reductions in computational time and have competitive regression performance on real-world data sets.

Regret Bounds for Online Kernel Selection in Continuous Kernel Space

Regret bounds of online kernel selection in a finite kernel set have been well studied, having at least an order O( √ NT) of magnitude after T rounds, where N is the number of candidate kernels. But it is still an unsolved problem to achieve sublinear regret bounds of online kernel selection in a continuous kernel space under different learning frameworks. In this paper, to represent different learning frameworks of online kernel selection, we divide online kernel selection approaches in a continuous kernel space into two categories according to the order of selection and training at each round. Then we construct a surrogate hypothesis space that contains all the candidate kernels with bounded norms and inner products, representing the continuously varying hypothesis space. Finally, we decompose the regrets of the proposed online kernel selection categories into different types of instantaneous regrets in the surrogate hypothesis space, and derive optimal regret bounds of order O( √ T) of magnitude under mild assumptions, independent of the cardinality of the continuous kernel space. Empirical studies verified the correctness of the theoretical regret analyses.

Implicit Kernel Attention

Attention computes the dependency between representations, and it encourages the model to focus on the important selective features. Attention-based models, such as Transformer and graph attention network (GAT), are widely utilized for sequential data and graph-structured data. This paper suggests a new interpretation and generalized structure of the attention in Transformer and GAT. For the attention in Transformer and GAT, we derive that the attention is a product of two parts: 1) the RBF kernel to measure the similarity of two instances and 2) the exponential of L2 norm to compute the importance of individual instances. From this decomposition, we generalize the attention in three ways. First, we propose implicit kernel attention with an implicit kernel function instead of manual kernel selection. Second, we generalize L2 norm as the Lp norm. Third, we extend our attention to structured multi-head attention. Our generalized attention shows better performance on classification, translation, and regression tasks.

An Improving Noise Immunity Algorithm for Multiple Kernel Boosting Using Noise Detection and Kernel Selection Technique

Focus on the problem that the Multiple Kernel Boosting(MKBoost) algorithm is sensitive to noise, a Multiple Kernel Boosting algorithm based on weight update and kernel selection is proposed. Firstly, the algorithm use the combined classification error rate of the previously selected classifier and the current classifier to be selected as the selection index of the kernel function in the weak classifier before the kernel of the base classifier is selected in each iteration; Secondly, in the weight update stage, a new weight update method is constructed by fusing the noise-detection and the average of weights in Multiple Kernel Boosting algorithm, which reduce the sensitivity to noise samples. Among the 8 of UCI data sets with varying levels of noise, the algorithm was compared with MKBoost-D1, MKBoost-D2, under the accuracy criteria, it performed better than traditional MKBoost algorithms. Experimental results show that the algorithm is able to effectively reduce the sensitivity of MKBoost to noise, and also has better robustness than traditional MKBoost algorithms.

AKRNet: A novel convolutional neural network with attentive kernel residual learning for feature learning of gearbox vibration signals

Vibration signals have been widely used for machine health monitoring and fault diagnosis. However, due to the complex working conditions, vibration signals collected from gearbox are generally nonlinear and non-stationary, which may contain multiple time scales and much noise. Considering these physical characteristics of vibration signals, in this paper, a novel deep neural network (DNN), attentive kernel residual network (AKRNet), is proposed for multi-scale feature learning from vibration signals. Firstly, multiple branches with different kernel widths are used to extract multi-scale features from vibration signals. Secondly, an attentive kernel selection is proposed to fuse the multiple branches features, where dynamic selection is developed to adaptively highlight the informative feature maps, while suppress the useless feature maps. Thirdly, an attentive residual block is developed to improve the feature learning performance, which not only can alleviate gradient vanishing, but also further enhances the impulsive features in feature maps. Finally, the effectiveness of AKRNet for feature learning of vibration signals is verified on two gearbox test rigs. The experimental results show that AKRNet has good capacity of feature learning on vibration signals. It performs better on gearbox fault diagnosis than other typical DNNs, e.g., stacked auto-encoders (SAE), one-dimensional CNN (1-D CNN) and residual network (ResNet).

AKSNet: A novel convolutional neural network with adaptive kernel width and sparse regularization for machinery fault diagnosis

Convolutional kernels have significant affections on feature learning of convolutional neural network (CNN). However, it is still a challenging problem to determine appropriate kernel width. Moreover, some features learned by convolutional layers are still redundant and noisy. Thus, adaptive selection of kernel width and feature selection of feature maps are key techniques to improve feature learning performance of CNNs. In this paper, a new deep neural network (DNN) model, adaptive kernel sparse network (AKSNet) is proposed to extract multi-scale fault features from one-dimensional (1-D) vibration signals. Firstly, an adaptive kernel selection method is developed, where multiple branches with different kernels are used to extract multi-scale features from vibration signals. Channel-wise attention is developed to fuse features generated by these kernels to obtain different informative scales. Secondly, a spatial attention is used for dynamic receptive field to focus on salient region of feature maps. Thirdly, a sparse regularization layer is embedded in the deep network to further filter noise and highlight impaction of the feature maps. Finally, two cases are adopted to verify effectiveness of AKSNet-based feature learning for bearing fault diagnosis. Experimental results show that AKSNet can effectively extract features from multi-channel vibration signals and then improves fault diagnosis performance of the classifier significantly. AKSNet shows better recognition performance in comparison with that of shallow neural networks and other typical DNNs.

A Two-Stream CNN Model with Adaptive Adjustment of Receptive Field Dedicated to Flame Region Detection

Convolutional neural networks (CNN) have yielded state-of-the-art performance in image segmentation. Their application in video surveillance systems can provide very useful information for extinguishing fire in time. The current studies mostly focused on CNN-based flame image classification and have achieved good accuracy. However, the research of CNN-based flame region detection is extremely scarce due to the bulky network structures and high hardware configuration requirements of the state-of-the-art CNN models. Therefore, this paper presents a two-stream convolutional neural network for flame region detection (TSCNNFlame). TSCNNFlame is a lightweight CNN architecture including a spatial stream and temporal stream for detecting flame pixels in video sequences captured by fixed cameras. The static features from the spatial stream and dynamic features from the temporal stream are fused by three convolutional layers to reduce the false positives. We replace the convolutional layer of CNN with the selective kernel (SK)-Shuffle block constructed by integrating the SK convolution into the deep convolutional layer of ShuffleNet V2. The SKnet blocks can adaptively adjust the size of one receptive field with the proportion of one region of interest (ROI) in it. The grouped convolution used in Shufflenet solves the problem in which the multi-branch structure of SKnet causes the network parameters to double with the number of branches. Therefore, the CNN network dedicated to flame region detection balances the efficiency and accuracy by the lightweight architecture, the temporal–spatial features fusion, and the advantages of the SK-Shuffle block. The experimental results, which are evaluated by multiple metrics and are analyzed from many angles, show that this method can achieve significant performance while reducing the running time.

SK-Unet: An Improved U-Net Model With Selective Kernel for the Segmentation of LGE Cardiac MR Images

In the clinical environment, myocardial infarction (MI) as one common cardiovascular disease is mainly evaluated using late gadolinium enhancement (LGE) cardiac magnetic resonance images (CMRIs). Accurate segmentation of the ventricles and the myocardium is a prerequisite for quantitative assessment of cardiac functions and disease progression. Performing the task using LGE images is, however, rather challenging due to heterogeneous image intensity distribution and lack of clear boundaries between adjacent organs and tissues. In this paper we propose a deep neural network method for automatic segmentation of the left ventricle (LV), right ventricle (RV), and left ventricular myocardium (LVM) from LGE CMRIs, which also leverages complementary information from cine and T2-weighted CMRIs if available. In the proposed method, termed SK-Unet, we augment the original U-Net model by adding a squeeze-and-excitation residual (SE-Res) module in the encoder and a selective kernel (SK) module in the decoder. The SE-Res module applies an attention mechanism to enhance informative feature extraction and suppress redundant ones. The SK module offers the ability to adaptively learn task-relevant multi-scale spatial features. We tested our method by participating in the MICCAI 2019 MS-CMRSeg challenge and achieved a mean dice score of 0.922 for LV segmentation, 0.827 for LVM, and 0.874 for RV. The results placed our method at the 1st place in the competition, and our accuracy of 0.827 also greatly surpasses the measured inter-observer agreement of 0.757 for manual segmentation of LVM in LGE CMRIs. The code accompanying our method is made available online at <uri>https://github.com/Xiyue-Wang/1st-in-MS-CMRSeg-2019</uri>.

Discriminating Between GARCH Models for Option Pricing by Their Ability to Compute Accurate VIX Measures

AbstractIn this article, we discuss the pricing performances of a large collection of GARCH models by questioning the global synergy between the choice of the affine/nonaffine GARCH specification, the use of competing alternatives to the Gaussian distribution, the selection of an appropriate pricing kernel, and the choice of different estimation strategies based on several sets of financial information. Furthermore, the study answers an important question in relation to the correlation between the performance of a pricing scheme and its ability to forecast VIX dynamics. VIX analysis clearly appears as a parsimonious first-stage filter to discard the worst GARCH option pricing models.

An advanced model of thermoelasticity with higher-order memory-dependent derivatives and dual time-delay factors

ABSTRACT This work gives some results and develops the derivation of memory dependence heat conduction model with dual phase-lags using the modified definition of the fractional derivative of Caputo and higher-order memory-dependent derivative (MDD). The introduced model was derived by improving the heat conduction equation replacing the conventional Fourier’s law and generalizing some of the previous models. As an application of the proposed model, we will consider an infinite thermoelastic long annular cylinder whose inner surface is thermally insulated and restricted, while the exterior surface is due to thermal shock and traction free. The numerical results of the physical fields are shown graphically against the position, and the effects of some basic parameters on them are estimated. Theoretical analysis and mathematical results confirm that the selection of an appropriate kernel, time delays and memory response can affect the turbulence in the thermal fields.

An Auto-Adjusted Kernel Method for Thermal Sharpening With Local and Object-Based Window Strategies

Thermal sharpening expands the application of land surface temperature due to the trade-off between spatial and temporal resolutions. Fixed kernels (FK) are widely applied in two window strategies: local window strategies (LWS) and object-based window strategies (OWS). However, the fixed regression kernel effect on LWS and OWS has rarely been considered in existing studies. Due to the heterogeneity among different windows, it is important to select suitable kernels for each window either in OWS or LWS. This article presents an auto-adjusted kernel (AAK) method to resolve this issue and examines certain simple kernel selection rules in thermal sharpening aiming to balance accuracy and efficiency. We test the AAK method with Landsat 8 data and compare it to the FK method with both OWS and LWS. The results reveal that the AAK method generally performs better than the FK method. Compared to the FK method, the AAK method improves the OWS accuracy by 0.283 K on average at three downscaling ratios, and the accuracy improvement increases with increasing downscaling ratio (from 3 to 9). Especially when the downscaling ratio reaches to 9, there is an evident improvement with 0.425 K of AAK. Moreover, the AAK method enhances the mean LWS accuracy by 0.179 K overall and decreases the difference between OWS and LWS. Furthermore, the AAK method increases the accuracy in specific areas and reduces extreme-value points. These findings indicate the potential of the AAK method in thermal sharpening with OWS and LWS, which resolves kernel selection problems.

Improved ECG-Derived Respiration Using Empirical Wavelet Transform and Kernel Principal Component Analysis.

Many methods have been developed to derive respiration signals from electrocardiograms (ECGs). However, traditional methods have two main issues: (1) focusing on certain specific morphological characteristics and (2) not considering the nonlinear relationship between ECGs and respiration. In this paper, an improved ECG-derived respiration (EDR) based on empirical wavelet transform (EWT) and kernel principal component analysis (KPCA) is proposed. To tackle the first problem, EWT is introduced to decompose the ECG signal to extract the low-frequency part. To tackle the second issue, KPCA and preimaging are introduced to capture the nonlinear relationship between ECGs and respiration. The parameter selection of the radial basis function kernel in KPCA is also improved, ensuring accuracy and a reduction in computational cost. The correlation coefficient and amplitude square coherence coefficient are used as metrics to carry out quantitative and qualitative comparisons with three traditional EDR algorithms. The results show that the proposed method performs better than the traditional EDR algorithms in obtaining single-lead-EDR signals.

A Self-Adaptive Multikernel Machine Based on Recursive Least-Squares Applied to Very Short-Term Wind Power Forecasting

Wind power has contributed significantly to the increase in electricity generation, but a decision-making tool capable of dealing with its variability and limited predictability is necessary. For this purpose, a novel self-adaptive approach for kernel recursive least-squares machines named multiple challengers is introduced in this work, which is successfully used to produce very short-term wind power forecasts at eight wind farms in Australia. The proposed method is based on a competitive tracking method, and the algorithm deals with some common difficulties of kernel methods, e.g., the increasing kernel matrix size associated with time and memory complexities and the overfitting problem. The proposed method always considers the new information received by the model, thus identifying changes in the time series, avoiding abrupt loss of information and maintaining a controlled number of examples since there is an adaptive selection of the active kernel. It works with the smallest dictionary possible, reducing the probability of overfitting. Five minute-ahead wind power forecasts are produced and evaluated in terms of point forecast skill scores and calibration. The results of the proposed method are compared with those provided by other kernel-based versions of the recursive least-squares algorithm, an online version of the extreme learning machine method, and the persistence time series model. An increase in the number of kernels used in the ensemble system can lead to better results when compared with those of single-kernel models.

Joint Generative Image Deblurring Aided by Edge Attention Prior and Dynamic Kernel Selection

Image deblurring is a classic and important problem in industrial fields, such as aviation photo restoration, object recognition in robotics, and autonomous vehicles. Blurry images in real‐world scenarios consist of mixed blurring types, such as a natural motion blurring owing to shaking of the camera. Fast deblurring does not deblur the entire image because it is not the best option. Considering the computational costs, it is also better to have an alternative kernel to deblur different objects at a high‐semantic level. To achieve better image restoration quality, it is also beneficial to combine the blurring category location and important structural information in terms of specific artifacts and degree of blurring. The goal of blind image deblurring is to restore sharpness from the unknown blurring kernel of an image. Recent deblurring methods tend to reconstruct prior knowledge, neglecting the influence of blur estimation and visual fidelity on image details and structure. Generative adversarial networks(GANs) have recently been attracting considerable attention from both academia and industry because GAN can perfectly generate new data with the same statistics as the training set. Therefore, this study proposes a generative neural architecture and an edge attention algorithm developed to restore vivid multimedia patches. Joint edge generation and image restoration techniques are designed to solve the low‐level multimedia retrieval. This multipath refinement fusion network (MRFNet) can not only perform deblurring of images directly but also individual the frames separately from videos. Ablation experiments validate that our generative adversarial network MRFNet performs better in joint training than in multimodel. Compared to other GAN methods, our two‐phase method exhibited state‐of‐the‐art performance in terms of speed and accuracy as well as has a significant visual improvement.