Deep Kernel Research Articles

A transformer fault diagnosis method based on EHBA-DHKELM-Adaboost model

Aiming at the problem of inaccurate transformer fault diagnosis in dissolved gas analysis, this paper proposes a novel diagnostic method that integrates an enhanced honey badger algorithm (EHBA) with an ensemble learning-based deep hybrid kernel extreme learning machine (DHKELM). First, kernel principal component analysis (KPCA) was deployed for feature fusion of the gas data, thus extracting more effective features. The DHKELM, combining polynomial and RBF kernel functions, was used as a base learning to build a powerful classifier with Adaboost framework. The EHBA introduces information sharing and firefly perturbation strategies based on HBA. This EHBA was harnessed to optimize the DHKELM’s critical parameters, establishing the EHBA-DHKELM-Adaboost transformer fault diagnosis model. Finally, the features garnered by KPCA were fed into the model, simulating and validating various fault diagnosis models. The findings reveal that EHBA-DHKELM-Adaboost achieves 98.75% diagnostic accuracy in transformer faults, surpassing other models.

Dead Reckoning in Emergency Vehicle Preemption System Using Deep Output Kernel Learning and Extended Kalman Filter

Emergency Vehicle Preemption (EVP) system plays an important role in reducing the response time of emergency vehicles by around 15–50%. Emergency Vehicle location estimation during Global Positioning System (GPS) outages is a challenging task in EVP. To address this issue, a location estimation scheme has been proposed by applying deep output kernel learning and extended Kalman filter using Inertial Measurement Unit (IMU) dead reckoning. The proposed scheme reduces the error in the estimation of velocity, attitude, and position by dynamically adapting the noise parameters of the extended Kalman filter. This system provides navigation solutions for emergency vehicles. Initially, four test routes were selected and the Position, velocity and attitude (pitch and roll angle) were measured and applied to the proposed scheme. Training and testing were conducted for the measured datasets. The performance of the proposed scheme was measured using five statistical methods during training and testing, and a comparison was made with other existing methods. The simulation results show that the proposed scheme performed well, in the four test routes. Finally, two different case studies were conducted using the proposed scheme and the performance was compared with three other methods. According to the results, the proposed scheme showed improvements in detection accuracy compared to the existing methods during GPS outages of 71.94% and 62.83% for Trajectory-1 and Trajectory-2, respectively.

Deep kernel methods learn better: from cards to process optimization

The ability of deep learning methods to perform classification and regression tasks relies heavily on their capacity to uncover manifolds in high-dimensional data spaces and project them into low-dimensional representation spaces. In this study, we investigate the structure and character of the manifolds generated by classical variational autoencoder (VAE) approaches and deep kernel learning (DKL). In the former case, the structure of the latent space is determined by the properties of the input data alone, while in the latter, the latent manifold forms as a result of an active learning process that balances the data distribution and target functionalities. We show that DKL with active learning can produce a more compact and smooth latent space which is more conducive to optimization compared to previously reported methods, such as the VAE. We demonstrate this behavior using a simple cards dataset and extend it to the optimization of domain-generated trajectories in physical systems. Our findings suggest that latent manifolds constructed through active learning have a more beneficial structure for optimization problems, especially in feature-rich target-poor scenarios that are common in domain sciences, such as materials synthesis, energy storage, and molecular discovery. The Jupyter Notebooks that encapsulate the complete analysis accompany the article.

Meta-learning with deep flow kernel network for few shot cross-domain remaining useful life prediction

Reliable prediction of the remaining useful life (RUL) is important for improving maintenance efficiency, equipment availability, and avoiding catastrophic accidents in complex industrial systems. Existing RUL prediction models have made some contribution, relying mainly on a large amount of degraded data with similar patterns or approximate distributions. However, in practical industrial systems, only a small amount of labeling data is usually available, which may also come from different devices and different working conditions, resulting in different distributions in the degraded data. This situation makes the existing RUL methods difficult to achieve satisfactory generalization performance. To address this challenge, this paper proposes a novel Meta-Learning with Deep Flow Kernel Network (MetaDFKN) model for RUL prediction under the few shot and cross-domain conditions. The model first learns kernel features in a data-driven manner and considers them as latent variables to improve the model’s representative ability of shared knowledge between tasks. Then, we introduce the conditional normalized flow technique to infer richer posterior distributions in the kernel features, which helps to obtain feature information with stronger discriminative power. Moreover, shared knowledge and task-specific information are integrated into the contextual inference process, which can mine the dependencies of related tasks and capture richer domain information. Finally, to evaluate the proposed model, we conduct extensive experiments on engine and bearings degradation data, and the results verify the superiority of the model.

A New Graph Autoencoder-Based Multi-level Kernel Subspace Fusion Framework for Single-cell Type Identification.

The advent of single-cell RNA sequencing (scRNA-seq) technology offers the opportunity to conduct biological research at the cellular level. Single-cell type identification based on unsupervised clustering is one of the fundamental tasks of scRNA-seq data analysis. Although many single-cell clustering methods have been developed recently, few can fully exploit the deep potential relationships between cells, resulting in suboptimal clustering. In this paper, we propose scGAMF, a graph autoencoder-based multi-level kernel subspace fusion framework for scRNA-seq data analysis. Based on multiple top feature sets, scGAMF unifies deep feature embedding and kernel space analysis into a single framework to learn an accurate clustering affinity matrix. First, we construct multiple top feature sets to avoid the high variability caused by single feature set learning. Second, scGAMF uses a graph autoencoder (GAEs) to extract deep information embedded in the data, and learn embeddings including gene expression patterns and cell-cell relationships. Third, to fully explore the deep potential relationships between cells, we design a multi-level kernel space fusion strategy. This strategy uses a kernel expression model with adaptive similarity preservation to learn a self-expression matrix shared by all embedding spaces of a given feature set, and a consensus affinity matrix across multiple top feature sets. Finally, the consensus affinity matrix is used for spectral clustering, visualization, and identification of gene markers. Extensive validation on real datasets shows that scGAMF achieves higher clustering accuracy than many popular single-cell analysis methods.

Data-Driven Joint Fault Diagnosis Based on RMK-ASSA and DBSKNet for Blast Furnace Iron-Making Process

Blast furnace iron-making process (BFIP) is one of the most critical procedures in the iron and steel industry where timely detection and accurate classification of faults have always been of core focus. However, the coupling effects of system’s nonlinear and nonstationary characteristics often cause process consistent underlying information to be buried, allowing accurate extraction to be a significant challenge. This also complicates the development of BFIP fault diagnosis model. Therefore, we propose a novel data-driven joint fault diagnosis strategy that employs regularized mutual kernel analytic stationary subspace analysis (RMK-ASSA) and deep broad stationary kernel network (DBSKNet) to eliminate this interference. To develop this method, we first construct an RMK-ASSA approach to address the poor modeling accuracy caused by standard analytic stationary subspace analysis (ASSA)’s inability to handle complex process nonlinearity. Global and local kernels are utilized to account for multiple nonlinearities in BFIP data. The weight of different nonlinear data is calculated by regularized principal component analysis, and the main information is imported into ASSA to obtain more robust and accurate modeling results by eliminating the interference of redundant noise. Subsequently, we design a DBSKNet-based classifier to implement the fault diagnosis task. This network further considers the nonlinearity by boosting kernel structure in depth and width while distinguishing the respective contributions of different kernels to fault diagnosis results. Finally, a double-layer loop parameter optimization algorithm is used for optimizing. Simulated cases and practical BFIP tests validate that RMK-ASSA eliminates the negative impact caused by nonstationary data and that the proposed joint fault diagnosis strategy outperforms other methods. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —BFIP’s nonlinear and nonstationary coupling properties pose unique challenges in eliminating distractions, constructing fault classifiers and accurately detecting process anomalies. To tackle these challenges, this paper proposes a joint fault diagnosis strategy based on RMK-ASSA and DBSKNet. RMK-ASSA effectively estimates nonlinear consistent features, while DBSKNet mines rich deep nonlinear information, accurately distinguishing variations in BFIP data under different working conditions. Experimental results demonstrate that this data-driven strategy can perform high-quality fault diagnosis, enabling field engineers to execute operations efficiently.

DK-Former: A Hybrid Structure of Deep Kernel Gaussian Process Transformer Network for Enhanced Traffic Sign Recognition

DK-Former: A Hybrid Structure of Deep Kernel Gaussian Process Transformer Network for Enhanced Traffic Sign Recognition

Are you sure it’s an artifact? Artifact detection and uncertainty quantification in histological images

Modern cancer diagnostics involves extracting tissue specimens from suspicious areas and conducting histotechnical procedures to prepare a digitized glass slide, called Whole Slide Image (WSI), for further examination. These procedures frequently introduce different types of artifacts in the obtained WSI, and histological artifacts might influence Computational Pathology (CPATH) systems further down to a diagnostic pipeline if not excluded or handled. Deep Convolutional Neural Networks (DCNNs) have achieved promising results for the detection of some WSI artifacts, however, they do not incorporate uncertainty in their predictions. This paper proposes an uncertainty-aware Deep Kernel Learning (DKL) model to detect blurry areas and folded tissues, two types of artifacts that can appear in WSIs. The proposed probabilistic model combines a CNN feature extractor and a sparse Gaussian Processes (GPs) classifier, which improves the performance of current state-of-the-art artifact detection DCNNs and provides uncertainty estimates. We achieved 0.996 and 0.938 F1 scores for blur and folded tissue detection on unseen data, respectively. In extensive experiments, we validated the DKL model on unseen data from external independent cohorts with different staining and tissue types, where it outperformed DCNNs. Interestingly, the DKL model is more confident in the correct predictions and less in the wrong ones. The proposed DKL model can be integrated into the preprocessing pipeline of CPATH systems to provide reliable predictions and possibly serve as a quality control tool.

A novel multi-class land use/land cover classification using deep kernel attention transformer for hyperspectral images

A novel multi-class land use/land cover classification using deep kernel attention transformer for hyperspectral images

Hybrid ML-MDKL feature subset selection and classification technique accompanied with rat swarm optimizer to classify the multidimensional breast cancer mammogram image

Breast cancer is a pathological condition characterized by the abnormal proliferation of cells inside the breast tissue. The biggest challenge at the moment is determining the most optimal subset from large datasets. Numerous methodologies have been presented for the purpose of selecting the most suitable subset from datasets with high dimensions. However, the outcomes have proven to be insufficient in effectively addressing extensive quantities of multidimensional datasets. This publication presents a proposal for an efficient approach of selecting numerous feature subsets and performing classification on Multidimensional Datasets (MDD). In this study, the Moment Invariant Wavelet Feature Extraction (MI-WFE) approach is employed for the purpose of feature extraction. The New Adaptive Hybrid Levy Flight based Cuckoo Search Optimization Algorithm is employed to identify the most pertinent subset of features. This study presents a novel approach, referred to as H-RS-LVCSO, which aims to enhance the local search capability and optimization speed of the algorithm. This is achieved through the hybridization of the Rat Swarm Optimizer with the levy flight based cuckoo search optimization technique. In this study, it is suggested a hybridized approach for classifying breast cancer, utilizing a Multilayer Multiple Deep Kernel Learning (ML-MDKL) Classifier in conjunction with the standard Support Vector Machine (SVM) Classifier. The simulation of the suggested method is conducted using the MATLAB software. The approach under consideration is evaluated in comparison to two pre-existing methods. The suggested method demonstrates an accuracy improvement of 10.58% compared to existing methods. Furthermore, it outperforms these methods by 17.5% in terms of accuracy. The proposed method exhibits a precision improvement of 6.52% and 14.23% compared to the existing methods.

Dynamic identification of coupler yaw angle of heavy haul locomotive: An optimal multi-task ELM-based method

The lateral instability of the couplers seriously threaten the operation safety of 20,000-tonne heavy-haul trains. It is essential to dynamically monitor the coupler yaw angle (CYA) in order to evaluate the operation stability of the train. However, the traditional direct measurement methods for CYA are hindered by the harsh service environment, making it difficult to maintain long-term stability in monitoring. To address these issues, within this paper a multi-task deep multiple kernel extreme learning machine (MT-DMKELM) is developed to quantitatively identify CYA based on the multi-channel relative displacements between the car body and frame (RDCFs). Firstly, the correlation of multi-channel RDCFs is analyzed based on the dynamics model, and the data fusion of strong correlation channels is achieved through an extreme learning machine auto-encoder. Then, a deep network for feature extraction is created by stacking numerous multiple kernel extreme learning machine auto-encoders. Further, the high-level abstract features are input to multiple kernel extreme learning machine for regression estimation of CYA. In addition, the optimal model parameters are obtained using the genetic mutation particle swarm optimization for MT-DMKELM to improve the CYA identification performance. Finally, both dynamics simulation and field test are conducted to confirm the capability of the proposed methods in CYA identification.

Deep Kernel Principal Component Analysis for multi-level feature learning

Principal Component Analysis (PCA) and its nonlinear extension Kernel PCA (KPCA) are widely used across science and industry for data analysis and dimensionality reduction. Modern deep learning tools have achieved great empirical success, but a framework for deep principal component analysis is still lacking. Here we develop a deep kernel PCA methodology (DKPCA) to extract multiple levels of the most informative components of the data. Our scheme can effectively identify new hierarchical variables, called deep principal components, capturing the main characteristics of high-dimensional data through a simple and interpretable numerical optimization. We couple the principal components of multiple KPCA levels, theoretically showing that DKPCA creates both forward and backward dependency across levels, which has not been explored in kernel methods and yet is crucial to extract more informative features. Various experimental evaluations on multiple data types show that DKPCA finds more efficient and disentangled representations with higher explained variance in fewer principal components, compared to the shallow KPCA. We demonstrate that our method allows for effective hierarchical data exploration, with the ability to separate the key generative factors of the input data both for large datasets and when few training samples are available. Overall, DKPCA can facilitate the extraction of useful patterns from high-dimensional data by learning more informative features organized in different levels, giving diversified aspects to explore the variation factors in the data, while maintaining a simple mathematical formulation.

Unfolded deep kernel estimation-attention UNet-based retinal image segmentation

Retinal vessel segmentation is a critical process in the automated inquiry of fundus images to screen and diagnose diabetic retinopathy. It is a widespread complication of diabetes that causes sudden vision loss. Automated retinal vessel segmentation can help to detect these changes more accurately and quickly than manual evaluation by an ophthalmologist. The proposed approach aims to precisely segregate blood vessels in retinal images while shortening the complication and computational value of the segmentation procedure. This can help to improve the accuracy and reliability of retinal image analysis and assist in diagnosing various eye diseases. Attention U-Net is an essential architecture in retinal image segmentation in diabetic retinopathy that obtained promising results in improving the segmentation accuracy especially in the situation where the training data and ground truth are limited. This approach involves U-Net with an attention mechanism to mainly focus on applicable regions of the input image along with the unfolded deep kernel estimation (UDKE) method to enhance the effective performance of semantic segmentation models. Extensive experiments were carried out on STARE, DRIVE, and CHASE_DB datasets, and the proposed method achieved good performance compared to existing methods.

Gaussian Process-Based Transfer Kernel Learning for Unsupervised Domain Adaptation

The discriminability and transferability of models are two important factors for the success of domain adaptation methods. Recently, some domain adaptation methods have improved models by adding a discriminant information extraction module. However, these methods need to carefully balance the discriminability and transferability of a model. To address this problem, we propose a new deep domain adaptation method, Gaussian Process-based Transfer Kernel Learning (GPTKL), which can perform domain knowledge transfer and improve the discrimination ability of the model simultaneously. GPTKL uses the kernel similarity between all samples in the source and target domains as a priori information to establish a cross-domain Gaussian process. By maximizing its likelihood function, GPTKL reduces the domain discrepancy between the source and target domains, thereby enhancing generalization across domains. At the same time, GPTKL introduces the deep kernel learning strategy into the cross-domain Gaussian process to learn a transfer kernel function based on deep features. Through transfer kernel learning, GPTKL learns a deep feature space with both discriminability and transferability. In addition, GPTKL uses cross-entropy and mutual information to learn a classification model shared by the source and target domains. Experiments on four benchmarks show that GPTKL achieves superior classification performance over state-of-the-art methods.

A Novel Deep Kernel Incremental Extreme Learning Machine Based on Artificial Transgender Longicorn Algorithm and Multiple Population Gray Wolf Optimization Methods

Redundant nodes in a kernel incremental extreme learning machine (KI-ELM) increase ineffective iterations and reduce learning efficiency. To address this problem, this study established a novel improved hybrid intelligent deep kernel incremental extreme learning machine (HI-DKIELM), which is based on a hybrid intelligent algorithm and a KI-ELM. First, a hybrid intelligent algorithm was established based on the artificial transgender longicorn algorithm and multiple population gray wolf optimization methods to reduce the parameters of hidden layer neurons and then to determine the effective number of hidden layer neurons. The learning efficiency of the algorithm was improved through the reduction of network complexity. Then, to improve the classification accuracy and generalization performance of the algorithm, a deep network structure was introduced to the KI-ELM to gradually extract the original input data layer by layer and realize high-dimensional mapping of data. The experimental results show that the number of network nodes of HI-DKIELM algorithm is obviously reduced, which reduces the network complexity of ELM and greatly improves the learning efficiency of the algorithm. From the regression and classification experiments, its CCPP can be seen that the training error and test error of the HI-DKIELM algorithm proposed in this paper are 0.0417 and 0.0435, which are 0.0103 and 0.0078 lower than the suboptimal algorithm, respectively. On the Boston Housing database, the average and standard deviation of this algorithm are 98.21 and 0.0038, which are 6.2 and 0.0003 higher than the suboptimal algorithm, respectively.

Counting manatee aggregations using deep neural networks and Anisotropic Gaussian Kernel

Manatees are aquatic mammals with voracious appetites. They rely on sea grass as the main food source, and often spend up to eight hours a day grazing. They move slow and frequently stay in groups (i.e. aggregations) in shallow water to search for food, making them vulnerable to environment change and other risks. Accurate counting manatee aggregations within a region is not only biologically meaningful in observing their habit, but also crucial for designing safety rules for boaters, divers, etc., as well as scheduling nursing, intervention, and other plans. In this paper, we propose a deep learning based crowd counting approach to automatically count number of manatees within a region, by using low quality images as input. Because manatees have unique shape and they often stay in shallow water in groups, water surface reflection, occlusion, camouflage etc. making it difficult to accurately count manatee numbers. To address the challenges, we propose to use Anisotropic Gaussian Kernel (AGK), with tunable rotation and variances, to ensure that density functions can maximally capture shapes of individual manatees in different aggregations. After that, we apply AGK kernel to different types of deep neural networks primarily designed for crowd counting, including VGG, SANet, Congested Scene Recognition network (CSRNet), MARUNet etc. to learn manatee densities and calculate number of manatees in the scene. By using generic low quality images extracted from surveillance videos, our experiment results and comparison show that AGK kernel based manatee counting achieves minimum Mean Absolute Error (MAE) and Root Mean Square Error (RMSE). The proposed method works particularly well for counting manatee aggregations in environments with complex background.

Transferable Deep Kernel Emulator for Probabilistic Load Margin Assessment With Topology Changes, Uncertain Renewable Generations and Loads

Transferable Deep Kernel Emulator for Probabilistic Load Margin Assessment With Topology Changes, Uncertain Renewable Generations and Loads

Few-shot remaining useful life prediction based on meta-learning with deep sparse kernel network

Predicting remaining useful life (RUL) of machinery is of vital importance to prognostics and health management. Reliable and accurate RUL prediction not only can reduce maintenance costs and increase machine availability but also even prevent catastrophic consequences. In reality, RUL predictions usually require numerous certain kinds of machine degradation data. However, complex operating conditions and safety issues may often result in fragmented data records generated, with very few complete samples being usable. To overcome the challenge of RUL prediction with limited data, this paper proposes a novel MetaDESK model that is based on meta-learning with deep sparse kernel network. The general idea is to train a sparse kernel with a variational posterior in a data-driven fashion, and then transfer it to a new few-shot RUL task via meta-knowledge. Specifically, we first incorporate a Gaussian Process into the model-agnostic meta-learning (MAML) framework and use variational inference to estimate latent variables as kernel features, which allows us to sample from a non-Gaussian distribution of the posterior. Then, the KL-divergence of sparse approximation is added to the kernel features as a regularization term through inference to reduce the overfitting problem. Also, to exploit the dependencies of the tasks we integrate both their shared knowledge and task-specific information into a contextual reasoning process, which is implemented by a bidirectional long short-term memory network. To evaluate our proposed model, we conduct extensive experiments using publicly available degradation data, and the results verify the model's effectiveness.

Explainability and human intervention in autonomous scanning probe microscopy

The broad adoption of machine learning (ML)-based autonomous experiments (AEs) in material characterization and synthesis requires strategies development for understanding and intervention in the experimental workflow. Here, we introduce and realize a post-experimental analysis strategy for deep kernel learning-based autonomous scanning probe microscopy. This approach yields real-time and post-experimental indicators for the progression of an active learning process interacting with an experimental system. We further illustrate how this approach can be applied to human-in-the-loop AEs, where human operators make high-level decisions at high latencies setting the policies for AEs, and the ML algorithm performs low-level, fast decisions. The proposed approach is universal and can be extended to other techniques and applications such as combinatorial library analysis.

A deep implicit memory Gaussian network for time series forecasting

In recent years, significant achievements have been made in time series forecasting using deep learning methods, particularly the Long Short-Term Memory Network (LSTM). However, time series often exhibit complex patterns and relationships like trends, seasonal patterns and irregularities, and LSTM networks fail to effectively strengthen long-term dependence in time series data, which may lead to inaccurate forecasting effect or performance degradation. Therefore, building a model to explore the temporal dependence within the time series data completely still remains a challenge. In this paper, we propose a novel Deep Implicit Memory Gaussian (DIMG) Network based on bidirectional deep memory kernel process and the implicit features enhancement method for time series forecasting. We first use the implicit features enhancement method to obtain the hidden features of the data according to the nonlinear mapping characteristics of encoder. Then, a new deep learning process called bidirectional deep memory kernel has been developed, which merges the structural properties of deep learning with the adaptability of kernel methods to capture intricate information and memory structures in sequential data. This process fully encapsulates the structure of Bi-LSTM and Gaussian process regression (GPR). Finally, the performance of the proposed model is evaluated on two real-world datasets. The experimental results verify that our model outperforms other reported methods in terms of prediction accuracy.