Kernel Embedding Research Articles

PQKELP: Projected Quantum Kernel Embedding based Link Prediction in dynamic networks

In dynamic networks, where the network’s topology is constantly changing, link prediction is a challenging issue. The two major challenges in link prediction within a time-varying network are accuracy and efficiency. Although random walk techniques have introduced promising embedding-based approaches, they fall short of optimization. Quantum computing, on the other hand, enhances performance in high-dimensional spaces, yet faces concerns about the limited efficiency of few qubit simulators. Addressing these issues, Projected Quantum Kernels (PQK) presents an elegant solution to achieve quantum advantage by simple quantum projection using the kernel trick, followed by a projection back to the classical state with relabeled data. In this work, we propose Projected Quantum Kernel Embedding based Link Prediction (PQKELP), a projected quantum kernel (PQK) approach on random walk embedding-based features to solve the link prediction problem. Thereby achieving a two-fold improvement by combining embedding generation and quantum projection, resulting in quantum-enhanced embedded features that enhance link prediction performance. Extensive experiments and analyses were done with a set of well-known random walk methods, i.e., Node2Vec, DeepWalk, and Walklets, also with six classical machine learning techniques and on five well-known dynamic network datasets. The results including several performance metrics like accuracy, AUC, F1-score, and precision show that our proposed model is better than traditional link prediction methods, classical machine learning approaches, and even the most cutting-edge methods available currently.

Machine learning in viscoelastic fluids via energy-based kernel embedding

The ability to measure differences in collected data is of fundamental importance for quantitative science and machine learning, motivating the establishment of metrics grounded in physical principles. In this study, we focus on the development of such metrics for viscoelastic fluid flows governed by a large class of linear and nonlinear stress models. To do this, we introduce energy-compatible families of kernel functions corresponding to a given viscoelastic stress model. Each kernel implicitly embeds flowfield snapshots into a Reproducing Kernel Hilbert Space (RKHS) in which distances and angles are computed and whose squared norm equals the total mechanical energy. Additionally, we present a solution to the preimage problem for these kernels, enabling accurate reconstruction of flowfields from their RKHS representations. Through numerical experiments on an unsteady viscoelastic lid-driven cavity flow, we demonstrate the utility of energy-compatible kernels for extracting energetically-dominant coherent structures in viscoelastic flows across a range of Reynolds and Weissenberg numbers. Specifically, the features extracted by Kernel Principal Component Analysis (KPCA) of flowfield snapshots using energy-compatible kernel functions yield reconstructions with superior accuracy in terms of mechanical energy compared to conventional methods such as ordinary Principal Component Analysis (PCA) with naïvely-defined state vectors or KPCA with ad-hoc choices of kernel functions. Our findings underscore the importance of principled choices of metrics in both scientific and machine learning investigations of complex fluid systems.

Detecting Critical Mismatched Driver Visual Attention During Lane Change: An Embedding Kernel Algorithm

Detecting Critical Mismatched Driver Visual Attention During Lane Change: An Embedding Kernel Algorithm

Kernel embedding of measures and low-rank approximation of integral operators

We describe a natural coisometry from the Hilbert space of all Hilbert-Schmidt operators on a separable reproducing kernel Hilbert space (RKHS)H\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\hbox { (RKHS)}\\, \\mathcal {H}$$\\end{document} and onto the RKHS G\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {G}$$\\end{document} associated with the squared-modulus of the reproducing kernel of H\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {H}$$\\end{document}. Through this coisometry, trace-class integral operators defined by general measures and the reproducing kernel of H\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {H}$$\\end{document} are isometrically represented as potentials in G\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {G}$$\\end{document}, and the quadrature approximation of these operators is equivalent to the approximation of integral functionals on G\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {G}$$\\end{document}. We then discuss the extent to which the approximation of potentials in RKHSs with squared-modulus kernels can be regarded as a differentiable surrogate for the characterisation of low-rank approximation of integral operators.

Fair large kernel embedding with relation-specific features extraction for link prediction

Knowledge graph embedding is a crucial technique for addressing the challenge of incomplete knowledge graphs, and convolutional neural networks are widely applied in this domain. However, convolutional neural networks face limitations in effectively capturing long-range dependencies between entities and relations in feature maps. Moreover, existing knowledge graph embedding models often adopt a uniform perspective on all relations within a knowledge graph, thereby overlooking the relation-specific features. To address the two issues, two novel knowledge graph embedding models, Large Kernel Embedding (LKE) and Large Kernel Embedding with Relation-specific Features Extraction (LKER), are proposed. The effectiveness of knowledge graph embedding is improved by integrating fair large kernel attention and introducing a parallel branch for relation-specific feature extraction in these models. Specifically, fair large kernel attention is incorporated into LKE, capturing long-range dependencies within the model. Based on LKE, LKER adds a parallel relationship-specific feature extraction branch to obtain more comprehensive relationship-specific feature information. The proposed models are extensively evaluated on five benchmark datasets, and the results show that the proposed models yield significant performance improvements in link prediction compared to classical and latest knowledge graph embedding models.

Few-shot segmentation using distribution adaption networks

Few-shot segmentation aims to achieve object-specific semantic segmentation in query images by conditioning on specific objects identified within support images. Previous methodologies predominantly relied on evaluating similarities between semantic-level prototypes of support features or comprehensive pixelwise support information and query images to inform predictions for queries. However, a significant domain discrepancy has been identified between the extracted query features and their support counterparts, resulting in a significant misalignment in the distribution of support and query features. As a result, direct analogies between support and query information cannot ensure the precision of query predictions. In this investigation, a distribution adaptation network is introduced, meticulously designed to learn discriminative representations and alleviate the distributional disparities between support features and query features. The adaptation network embeds both support and query features into Reproducing Kernel Hilbert Spaces (RKHS) and optimally aligns their distributions. The distributional gap is effectively bridged through optimal kernel embedding alignment, facilitated by a multiple kernel learning strategy. Extensive experimentation on datasets PASCAL-5i, COCO-20i and FSS-1000 unequivocally demonstrates that the proposed Kernel-Aligned Distribution Network (KADN) achieves state-of-the-art performance.

On-the-move heterogeneous face recognition in frequency and spatial domain using sparse representation.

Heterogeneity of a probe image is one of the most complex challenges faced by researchers and implementers of current surveillance systems. This is due to existence of multiple cameras working in different spectral ranges in a single surveillance setup. This paper proposes two different approaches including spatial sparse representations (SSR) and frequency sparse representation (FSR) to recognize on-the-move heterogeneous face images with database of single sample per person (SSPP). SCface database, with five visual and two Infrared (IR) cameras, is taken as a benchmark for experiments, which is further confirmed using CASIA NIR-VIS 2.0 face database with 17580 visual and IR images. Similarity, comparison is performed for different scenarios such as, variation of distances from a camera and variation in sizes of face images and various visual and infrared (IR) modalities. Least square minimization based approach for finding the solution is used to match face images as it makes the recognition process simpler. A side by side comparison of both the proposed approaches with the state-of-the-art, classical, principal component analysis (PCA), kernel fisher analysis (KFA) and coupled kernel embedding (CKE) methods, along with modern low-rank preserving projection via graph regularized reconstruction (LRPP-GRR) method, is also presented. Experimental results suggest that the proposed approaches achieve superior performance.

TOT：Topology-Aware Optimal Transport for Multimodal Hate Detection

Multimodal hate detection, which aims to identify the harmful content online such as memes, is crucial for building a wholesome internet environment. Previous work has made enlightening exploration in detecting explicit hate remarks. However, most of their approaches neglect the analysis of implicit harm, which is particularly challenging as explicit text markers and demographic visual cues are often twisted or missing. The leveraged cross-modal attention mechanisms also suffer from the distributional modality gap and lack logical interpretability. To address these semantic gap issues, we propose TOT: a topology-aware optimal transport framework to decipher the implicit harm in memes scenario, which formulates the cross-modal aligning problem as solutions for optimal transportation plans. Specifically, we leverage an optimal transport kernel method to capture complementary information from multiple modalities. The kernel embedding provides a non-linear transformation ability to reproduce a kernel Hilbert space (RKHS), which reflects significance for eliminating the distributional modality gap. Moreover, we perceive the topology information based on aligned representations to conduct bipartite graph path reasoning. The newly achieved state-of-the-art performance on two publicly available benchmark datasets, together with further visual analysis, demonstrate the superiority of TOT in capturing implicit cross-modal alignment.

Kernel embedding transformation learning for graph matching

• Developing a kernel embedding transformation learning model for graph matching. • Combining the kernelized unary alignment and local structure alignment into a joint framework. • Proposing an effective optimization algorithm to solve the joint framework. • Improving the matching performance significantly in the graph matching tasks with deformation and rotation variations. Graph matching , which aims to establish correspondences between two geometrical graphs, is a general and powerful tool for pattern recognition and computer vision . However, many factors degrade the matching accuracy. The graph structure suffering from deformation and rotation variations is a key issue in the process of matching. In this work, we propose a joint framework in the reproducing kernel Hilbert space (RKHS) for graph matching with deformation and rotation variations, which incorporates the kernelized unary alignment and local structure alignment into a joint framework. Specifically, the proposed method is able to enhance the node to node correspondence and the edge to edge correspondence and avoids the effect of deformation and rotation by maximizing the similarities between the source graph and the transformed target graph in the reproducing kernel Hilbert space. Meanwhile, an effective algorithm is presented to solve the joint framework. Comprehensive discussion, involving convergence analysis and parameter sensitive analysis , are as well proposed. Promising experimental results in the variety of graph matching tasks such as deformation and rotation are provided to evidence the superiority of the proposed method.

High-dimensional variable screening through kernel-based conditional mean dependence

This article proposes a model-free feature screening method for ultrahigh-dimensional data. The proposed method is built on the called kernel-based conditional mean dependence, which is defined based on the reproducing kernel embedding and the Hilbert–Schmidt norm of a tensor operator. Theoretically, both sure screening and rank consistency properties are established under weak assumptions. Through an extensive simulation study as well as a real data analysis, it is illustrated that the statistics generated by other kernels in distance kernel family are more sensitive to feature screening in ultra-high dimensions.

Prepare for the Worst, Hope for the Best: Active Robust Learning On Distributions.

In recent years, many learning systems have been developed for higher level forms of data, such as learning on distributions in which each example itself is a distribution. This article proposes active robust learning on distributions. In learning on distributions, there is no access to distributions themselves but rather access is through a sample drawn from a distribution. Therefore, similar to robust learning, any estimates of examples are inexact. In order to address these difficulties, we provide an upper bound on the risk of the classifier in the next stage of active learning, where the size of the labeled dataset increases. Based on this upper bound, we propose probabilistic minimax active learning (PMAL) as a general multiclass active learning method that is easy to use in many Bayesian settings, which provably selects an example with knowledge of its label minimizing the expected risk. We present an efficient approximation of the objective with a known error bound to deal with the intractability of the proposed method for active robust learning. Here, we face a nonconvex problem, which we solve by means of a related convex problem with a bound on the norm of the difference between their solutions. To utilize the information about the estimates of distributions, we propose active robust learning on the distributions method based on learning the kernel embedding of distributions by a recent Bayesian method. The experiments demonstrate the effectiveness of the resulting method on a set of synthetic and real-world distributional datasets.

Deep embedding kernel mixture networks for conditional anomaly detection in high-dimensional data

In various industrial problems, sensor data are often used to detect the abnormal state of manufacturing systems. Sensor data are sometimes influenced by contextual variables that are not related to the system health status and may exhibit different behaviours depending on their values, even if the system is in a normal condition. In this case, a conditional anomaly detection method should be used to consider the effects of contextual variables. In this study, we propose a conditional anomaly detection method, particularly for high-dimensional and complex data, using a deep embedding kernel mixture network. The proposed method comprises embedding and kernel mixture networks. The embedding network learns low-dimensional embeddings from high-dimensional data, and the kernel mixture network models the distribution of the learned embeddings conditional on contextual variables. The two networks enable a flexible estimation of conditional density using the high expressive power of deep neural networks. The two networks are trained simultaneously such that the high-dimensional data are embedded into a low-dimensional space, to assist conditional density estimation. The effectiveness of the proposed model is demonstrated using real data examples from the UCI repository and a case study from a tire company.

Distance metric learning with local multiple kernel embedding

Distance metric learning with local multiple kernel embedding

Feature selection for kernel methods in systems biology.

The substantial development of high-throughput biotechnologies has rendered large-scale multi-omics datasets increasingly available. New challenges have emerged to process and integrate this large volume of information, often obtained from widely heterogeneous sources. Kernel methods have proven successful to handle the analysis of different types of datasets obtained on the same individuals. However, they usually suffer from a lack of interpretability since the original description of the individuals is lost due to the kernel embedding. We propose novel feature selection methods that are adapted to the kernel framework and go beyond the well-established work in supervised learning by addressing the more difficult tasks of unsupervised learning and kernel output learning. The method is expressed under the form of a non-convex optimization problem with a ℓ1 penalty, which is solved with a proximal gradient descent approach. It is tested on several systems biology datasets and shows good performances in selecting relevant and less redundant features compared to existing alternatives. It also proved relevant for identifying important governmental measures best explaining the time series of Covid-19 reproducing number evolution during the first months of 2020. The proposed feature selection method is embedded in the R package mixKernel version 0.8, published on CRAN. Installation instructions are available at http://mixkernel.clementine.wf/.

Transfer learning for regression via latent variable represented conditional distribution alignment

Since labelled data are expensive to generate both computationally and experimentally, how to establish data-driven models with limited data has become an important challenge in various engineering and scientific applications. Transfer learning has been used to improve the performance of many tasks by leveraging rich labelled data from related domains. This paper focuses on the transfer learning problem for regression under the situation of conditional distribution shift, which is a common scenario in manufacturing industry and has not been fully studied. Hence, we first propose a Latent variable represented Conditional Distribution Alignment method (LCDA), which exploits the low-dimensional latent representation to describe the global conditional distribution discrepancy. Then the properties of the latent variables are optimised by matching the kernel embedding conditional distribution between domains, so that the global distribution discrepancy can be aligned with the residual function generated by the latent variables. By combining domain priors with machine learning, the proposed method provides a potential direction to reduce labelling consumption for the intelligent manufacturing. Series of experiments on two industrial applications are conducted to validate the effectiveness and analyse the characteristics of the proposed method.

Adaptive Classification Using Incremental Linearized Kernel Embedding

This paper considers the problem of adaptive classification for performing pattern discrimination in varying conditions when new data arrives. A new efficient method is presented to incrementally update features in a Nyström-approximated linearized kernel embedding (LKE). Our method leverages a fast eigendecomposition algorithm for symmetric Arrowhead matrices. The proposed method can also be applied to kernel principal component analysis (KPCA) or similar problems. A mechanism is proposed which allows the incremental linearized kernel embedding to be used for updating of dictionaries in a sparse representation-based classification algorithm. The method is based on transporting the dictionaries into the embedding expanded with new data points and avoids the need to learn new dictionary matrices every time new data becomes available. The effectiveness of the developed methods is illustrated on two handwritten digit image data sets namely MNIST and USPS. Classification performance before and after sequential embedding updates is evaluated and compared. Comparisons are also made between our incremental LKE algorithm and the conventional approach to updating the empirical kernel map in terms of their computational requirements and numerical stability.

Robust multimodal discrete hashing for cross-modal similarity search

Hashing technology improves the search efficiency and reduces the storage space of data. However, building an effective modal with unsupervised cross modal retrieval and generating efficient binary code is still a challenging task, considering of some issues needed to be further discussed and researched for unsupervised multimodal hashing. Most of the existing methods ignore the discrete restriction, and manually or experientially determine the weights of each modality. These limitations may significantly reduce the retrieval accuracy of unsupervised cross-modal hashing methods. To solve these problems, we propose a robust hash modal that can efficiently learn binary code by employing a flexible and noise-resistant ℓ2,1-loss with nonlinear kernel embedding. In addition, we introduce an intermediate state mapping that facilitate later modal optimization to measure the loss between the hash codes and the intermediate states. Experiments on several public multimedia retrieval datasets validate the superiority of the proposed method from various aspects.

Kernel Embedding Based Variational Approach for Low-Dimensional Approximation of Dynamical Systems

Abstract Transfer operators such as Perron–Frobenius and Koopman operator play a key role in modeling and analysis of complex dynamical systems, which allow linear representations of nonlinear dynamics by transforming the original state variables to feature spaces. However, it remains challenging to identify the optimal low-dimensional feature mappings from data. The variational approach for Markov processes (VAMP) provides a comprehensive framework for the evaluation and optimization of feature mappings based on the variational estimation of modeling errors, but it still suffers from a flawed assumption on the transfer operator and therefore sometimes fails to capture the essential structure of system dynamics. In this paper, we develop a powerful alternative to VAMP, called kernel embedding based variational approach for dynamical systems (KVAD). By using the distance measure of functions in the kernel embedding space, KVAD effectively overcomes theoretical and practical limitations of VAMP. In addition, we develop a data-driven KVAD algorithm for seeking the ideal feature mapping within a subspace spanned by given basis functions, and numerical experiments show that the proposed algorithm can significantly improve the modeling accuracy compared to VAMP.

A kernel-based measure for conditional mean dependence

A novel metric, called kernel-based conditional mean dependence (KCMD), is proposed to measure and test the departure from conditional mean independence between a response variable Y and a predictor variable X, based on the reproducing kernel embedding and the Hilbert-Schmidt norm of a tensor operator. The KCMD has several appealing merits. It equals zero if and only if the conditional mean of Y given X is independent of X, i.e. E(Y|X)=E(Y) almost surely, provided that the employed kernel is characteristic; it can be used to detect all kinds of conditional mean dependence with an appropriate choice of kernel; it has a simple expectation form and allows an unbiased empirical estimator. A class of test statistics based on the estimated KCMD is constructed, and a wild bootstrap test procedure to the conditional mean independence is presented. The limit distributions of the test statistics and the bootstrapped statistics under null hypothesis, fixed alternative hypothesis and local alternative hypothesis are given respectively, and a data-driven procedure to choose a suitable kernel is suggested. Simulation studies indicate that the tests based on the KCMD have close powers to the tests based on martingale difference divergence in monotone dependence, but excel in the cases of nonlinear relationships or the moment restriction on X is violated. Two real data examples are presented for the illustration of the proposed method.

On the noise estimation statistics

Learning with noisy labels has attracted much attention during the past few decades. A fundamental problem is how to estimate noise proportions from corrupted data. Previous studies on this issue resort to the estimations of class distributions, conditional distributions, or the kernel embedding of distributions. In this paper, we present another simple and effective approach for noise estimation. The basic idea is to utilize the first- and second-order statistics of observed data, and the positive semi-definiteness of covariance matrices. Then, an upper bound on noise estimation is provided without additional assumptions over data distribution. Based on this idea and using the locality property of random noise, we develop the Noise Estimation Statistics with Clusters (NESC) method, which firstly clusters the corrupted data by k-means algorithm, and then makes noise estimation from clusters based on the first- and second-order statistics. We present the existence, uniqueness and convergence analysis of our noise estimation, and empirical studies verify the effectiveness of the NESC method.