Hilbert-Schmidt Independence Criterion Research Articles

One step at a time: A statistical approach for distinguishing mediators, confounders, and colliders using direction dependence analysis.

In observational data, understanding the causal link when estimating the causal effect of an independent variable (x) on a dependent variable (y) often requires researchers to identify the role of a third variable in the x → y relationship. Mediation, confounding, and colliding are three key third-variable effects that yield different theoretical and methodological implications for drawing causal conclusions. Commonly used covariance-based statistical methods, such as linear regression and structural equation modeling, cannot distinguish these effects in practice, however. In this study, we introduce a statistical approach for distinguishing mediators, confounders, colliders, and potential M-bias structures that uses higher-order moment information from the data. We propose a two-step procedure that uses the Hilbert-Schmidt independence criterion within the direction dependence analysis framework. Results from Monte Carlo simulations show that our proposed approach accurately recovers the true data-generating process of the third variable. We provide an empirical example to demonstrate the application of our proposed approach in psychological research. Finally, we discuss implications and future directions of our work. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Testing independence of functional variables by an Hilbert–Schmidt independence criterion estimator

We propose a method for testing independence of functional variables by using an estimator of the Hilbert–Schmidt Independence Criterion obtained from an appropriate modification of the usual estimator. We get asymptotic normality of this estimator both under independence hypothesis and under the alternative hypothesis. A simulation study that allows to compare the proposed test to an existing one is provided.

Consistency–exclusivity guided unsupervised multi-view feature selection

Unsupervised multi-view feature selection (UMFS) is an effective dimension reduction for multi-view data. It aims to obtain the important feature subset from multi-view data, which can significantly minimize the impacts of noises, outliers and redundancy. Although previous UMFS methods achieve remarkable achievements, they fail to fully take into account the consistency or the exclusivity hidden in multi-view data. To address this, this article presents a consistency–exclusivity guided unsupervised multi-view feature selection (CE-UMFS) method. Specifically, we design a multi-view matrix factorization model to simultaneously explore the consistency and exclusivity in multi-view data. Meanwhile, we employ the Hilbert–Schmidt independence criterion (HSIC) to preserve the exclusivity of different views. Furthermore, we impose a nuclear norm on the consistent representation matrix to explore the consistency across views. At last, promising experimental results demonstrate the superiority of the proposed method compared with some state-of-the-art methods.

Multi-target HSIC-Lasso via exploiting target correlations

Multi-target regression (MTR) has been widely applied due to its compact structure and better computational efficiency in learning multiple related tasks, and has received extensive attention in machine learning, while great challenge arises from handling complex nonlinear relationships in high-dimensional data. In this study, we propose a novel approach to nonlinear dimensionality reduction in MTR. Specifically, the Hilbert–Schmidt Independence Criterion Lasso (HSIC-Lasso) is introduced to captures nonlinear dependencies between input–output and inter-target relationships in a unified framework. Additionally, sparse learning is incorporated to effectively eliminate redundant and irrelevant features. The proposed method is guaranteed to find an optimal subset of maximally predictive features with minimal redundancy, and explores target correlations by treating the targets as input features, which achieves a better learning of the target dependencies and thus improves the effectiveness of the feature selection. Extensive experiments on twelve public data sets have shown that the proposed methods outperform the compared methods in feature selection for the MTR problem.

Optimizing Graph Neural Network With Multiaspect Hilbert-Schmidt Independence Criterion.

The graph neural network (GNN) has demonstrated its superior power in various data mining tasks and has been widely applied in diversified fields. The core of GNN is the aggregation and combination functions, and mainstream GNN studies focus on the enhancement of these functions. However, GNNs face a common challenge, i.e., useless features contained in neighbor nodes may be integrated into the target node during the aggregation process. This leads to poor node embedding and undermines downstream tasks. To tackle this problem, this article proposes a novel GNN optimization framework GNN-MHSIC by introducing the nonparametric dependence method Hilbert-Schmidt independence criterion (HSIC) under the guidance of information bottleneck. HSIC is utilized to guide the information propagation among layers of a GNN from multiaspect views. GNN-MHSIC aims to achieve three main objectives: 1) minimizing the HSIC between the input features and the propagation layers; 2) maximizing the HSIC between the propagation layers and the ground truth; and 3) minimizing the HSIC between the propagation layers. With a multiaspect design, GNN-MHSIC can minimize the propagation of redundant information while preserving relevant information about the target node. We prove GNN-MHSIC's finite upper and lower bounds theoretically and evaluate it experimentally with four classic GNN models, including the graph convolutional network, the graph attention network (GAT), the heterogeneous GAT, and the heterogeneous graph (HG) propagation network on three widely used HGs. The results illustrate the usefulness and performance of GNN-MHSIC.

Discriminative multi-label feature selection with adaptive graph diffusion

Feature selection can alleviate the problem of the curse of dimensionality by selecting more discriminative features, which plays an important role in multi-label learning. Recently, embedded feature selection methods have received increasing attentions. However, most existing methods learn the low-dimensional embeddings under the guidance of the local structure between the original instance pairs, thereby ignoring the high-order structure between instances and being sensitive to noise in the original features. To address these issues, we propose a feature selection method named discriminative multi-label feature selection with adaptive graph diffusion (MFS-AGD). Specifically, we first construct a graph embedding learning framework equipped with adaptive graph diffusion to uncover a latent subspace that preserves the higher-order structure information between four tuples. Then, the Hilbert–Schmidt independence criterion (HSIC) is incorporated into the embedding learning framework to ensure the maximum dependency between the latent representation and labels. Benefiting from the interactive optimization of the feature selection matrix, latent representation and similarity graph, the selected features can accurately explore the higher-order structural and supervised information of data. By further considering the correlation between labels, MFS-AG is extended to a more discriminative version,i.e., LMFS-AG. Extensive experimental results on various benchmark data sets validate the advantages of the proposed MFS-AGD and LMFS-AGD methods.

Kernel-Based Independence Tests for Causal Structure Learning on Functional Data.

Measurements of systems taken along a continuous functional dimension, such as time or space, are ubiquitous in many fields, from the physical and biological sciences to economics and engineering. Such measurements can be viewed as realisations of an underlying smooth process sampled over the continuum. However, traditional methods for independence testing and causal learning are not directly applicable to such data, as they do not take into account the dependence along the functional dimension. By using specifically designed kernels, we introduce statistical tests for bivariate, joint, and conditional independence for functional variables. Our method not only extends the applicability to functional data of the Hilbert-Schmidt independence criterion (hsic) and its d-variate version (d-hsic), but also allows us to introduce a test for conditional independence by defining a novel statistic for the conditional permutation test (cpt) based on the Hilbert-Schmidt conditional independence criterion (hscic), with optimised regularisation strength estimated through an evaluation rejection rate. Our empirical results of the size and power of these tests on synthetic functional data show good performance, and we then exemplify their application to several constraint- and regression-based causal structure learning problems, including both synthetic examples and real socioeconomic data.

Kernel-based joint independence tests for multivariate stationary and non-stationary time series.

Multivariate time-series data that capture the temporal evolution of interconnected systems are ubiquitous in diverse areas. Understanding the complex relationships and potential dependencies among co-observed variables is crucial for the accurate statistical modelling and analysis of such systems. Here, we introduce kernel-based statistical tests of joint independence in multivariate time series by extending the d-variable Hilbert-Schmidt independence criterion to encompass both stationary and non-stationary processes, thus allowing broader real-world applications. By leveraging resampling techniques tailored for both single- and multiple-realization time series, we show how the method robustly uncovers significant higher-order dependencies in synthetic examples, including frequency mixing data and logic gates, as well as real-world climate, neuroscience and socio-economic data. Our method adds to the mathematical toolbox for the analysis of multivariate time series and can aid in uncovering high-order interactions in data.

Identification of DNA N4-methylcytosine Sites via Multiview Kernel Sparse Representation Model

Identifying DNA N4-methylcytosine (4mC) sites is of great significance in biological research, such as chromatin structure, DNA stability, DNA-protein interaction and controlling gene expression. However, the traditional sequencing technology to identify 4mC sites is very time-consuming. In order to detect 4mC sites, we develop a multi-view learning method for achieving more effectively via merging multiple feature spaces. Furthermore, we think about whether the multi-view learning method can improve the across species classification ability by fusing data of multiple species. In our study, we propose a multi-view Laplacian kernel sparse representation-based classifier, called MvLapKSRC-HSIC. First, we make use of three feature extraction methods (PSTNP, NCP, DPP) to extract the DNA sequence features. MvLapKSRC-HSIC uses a kernel sparse representation-based classifier with graph regularization. In order to maintain the independence between various views, we add a multi-view regularization term constructed by Hilbert-Schmidt independence criterion (HSIC). In the experiments, MvLapKSRC-HSIC is applied on six datasets, so as to compare with other popular methods in single species and cross-species experiments. All experimental results show that MvLapKSRC-HSIC is superior to other outstanding methods on both single species and cross-species. Importantly, MvLapKSRC-HSIC can identify a series of potential DNA 4mC sites, which have not yet been experimentally evaluate on multiple species and merit further research.

Test of conditional independence in factor models via Hilbert–Schmidt independence criterion

This work is concerned with testing conditional independence under a factor model setting. We propose a novel multivariate test for non-Gaussian data based on the Hilbert–Schmidt independence criterion (HSIC). Theoretically, we investigate the convergence of our test statistic under both the null and the alternative hypotheses, and devise a bootstrap scheme to approximate its null distribution, showing that its consistency is justified. Methodologically, we generalize the HSIC-based independence test approach to a situation where data follow a factor model structure. Our test requires no nonparametric smoothing estimation of functional forms including conditional probability density functions, conditional cumulative distribution functions and conditional characteristic functions under the null or alternative, is computationally efficient and is dimension-free in the sense that the dimension of the conditioning variable is allowed to be large but finite. Further extension to nonlinear, non-Gaussian structure equation models is also described in detail and asymptotic properties are rigorously justified. Numerical studies demonstrate the effectiveness of our proposed test relative to that of several existing tests.

FAM3L: Feature-Aware Multi-Modal Metric Learning for Integrative Survival Analysis of Human Cancers.

Survival analysis is to estimate the survival time for an individual or a group of patients, which is a valid solution for cancer treatments. Recent studies suggested that the integrative analysis of histopathological images and genomic data can better predict the survival of cancer patients than simply using single bio-marker, for different bio-markers may provide complementary information. However, for the given multi-modal data that may contain irrelevant or redundant features, it is still challenge to design a distance metric that can simultaneously discover significant features and measure the difference of survival time among different patients. To solve this issue, we propose a Feature-Aware Multi-modal Metric Learning method (FAM3L), which not only learns the metric for distance constraints on patients' survival time, but also identifies important images and genomic features for survival analysis. Specifically, for each modality of data, we firstly design one feature-aware metric that can be decoupled into a traditional distance metric and a diagonal weight for important feature identification. Then, in order to explore the complex correlation across multiple modality data, we apply Hilbert-Schmidt Independence Criterion (HSIC) to jointly learn multiple metrics. Finally, based on the learned distance metrics, we apply the Cox proportional hazards model for prognosis prediction. We evaluate the performance of our proposed FAM3L method on three cancer cohorts derived from The Cancer Genome Atlas (TCGA), the experimental results demonstrate that our method can not only achieve superior performance for cancer prognosis, but also identify meaningful image and genomic features correlating strongly with cancer survival.

Multi-kernel learning for multi-label classification with local Rademacher complexity

Multi-label classification aims to construct prediction models from input space to output space for multi-label datesets. However, the feature space of multi-label dataset and the hypothesis space of classifier are always complex. In order to obtain high performance multi-kernel learning algorithms, the core problem is to design algorithms that compress both hypothesis space and feature space. In this paper, by combining the local Rademacher complexity and Hilbert-Schmidt independence criterion (HSIC), an effective multi-kernel learning algorithm for multi-label classification is proposed to compress the feature space and hypothesis space simultaneously. Based on the tail sum of eigenvalues of the integral operator corresponding to kernels, the upper bound of local Rademacher complexity of linear functions class in the feature space is determined. The Hilbert-Schmidt independence criterion is applied to maximize the correlation between the convex combination of the input kernel matrices and the ideal kernel matrix in the label space. Finally, the Laplace regularization model is built by controlling the tail sum of eigenvalues of the integral operator corresponding to kernels of this criterion. Therefore, the above process obtains a combined kernel, simultaneously by sorting the coefficients of the combined kernel, a feature selection algorithm can be constructed. On the basis, the combined kernel can also be used to design a new binary classifier for multi-label classification. The experimental results verify that our proposed multi-kernel learning algorithms can effectively compress the hypothesis space, while the feature selection algorithms can effectively compress the feature space.

Disentangled Graph Contrastive Learning With Independence Promotion

Self-supervised learning for graph neural networks has attracted considerable attention and shows notable successes in graph representation learning. However, the formation of a real-world graph typically arises from highly complex interactions of many latent factors. The existing self-supervised learning methods for GNNs are inherently holistic and neglect the entanglement of the latent factors, resulting in suboptimal learned representations for downstream tasks and difficult to be interpreted. Learning disentangled graph representations with self-supervised learning poses great challenges and remains largely ignored by the existing literature. In this paper, we introduce Independence Promoted Disentangled Graph Contrastive Learning ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">IDGCL</b> ) method, which can learn disentangled graph-level representations with self-supervision. In particular, we first identify the latent factors of the input graph and derive its factorized representations. Then we propose a factor-wise discrimination objective in a contrastive learning manner, which can force the factorized representations to independently reflect the expressive information from different latent factors. To further promote the independence between the representations, we employ the Hilbert-Schmidt Independence Criterion to eliminate the dependence among different representations, which is effectively integrated into the self-supervised framework as a regularizer. Extensive experiments on synthetic and real-world datasets demonstrate the superiority of our method against several state-of-the-art baselines.

Statistical Methods for the Study of Computer Experiment Failures: Application to a Fuel-Coolant-Interaction Simulation Code

In the framework of risk assessment in nuclear accidents, simulation tools are widely used to understand and model physical phenomena. These simulation tools take into account a large number of uncertain input parameters. We often use Monte Carlo–type methods to explore their range of variation: The input space is randomly sampled, and a code run is performed on each sampled point. However, some of these code runs may fail to converge. Analyzing these code failures to understand which of the inputs have the most influence on them leads to a better understanding of how the code works. It also intends to improve the robustness of the simulation software and code computations. For this purpose, we propose two complementary approaches performing a statistical analysis of the code failures. The first approach is based on goodness-of-fit tests and compares conditional probability distributions according to code failures to a reference one. A second approach, based on a dependence measure named the Hilbert-Schmidt Independence Criterion, provides another way to measure the global dependence between the inputs and the code failures. The development of this methodology is carried out in the context of severe nuclear accidents. More especially, the presented methods are applied for the study of the simulation code MC3D, which simulates the fuel-coolant interaction in a severe nuclear accident context.

MVMA-GCN: Multi-view multi-layer attention graph convolutional networks

The accuracy of graph representation learning is highly dependent on the precise characterization of node relationships. However, representing the complex and diverse networks in the real world using a single type of node or link is challenging, often resulting in incomplete information. Moreover, different types of nodes and links convey rich information, which makes it difficult to design a graph network that can integrate diverse links. This paper introduces a novel multi-view and multi-layer attention model designed to optimize node embeddings for semi-supervised node classification. The proposed model exploits various types of inter-node links and employs the Hilbert–Schmidt independence criterion to maximize the dissimilarity between distinct node relationships. Furthermore, the multi-layer attention mechanism is used to discern the impact of different neighboring nodes and relationships between various node relationships. The performance of the proposed model, MVMA-GCN, was assessed on numerous real-world multi-view datasets. It was observed that MVMA-GCN consistently outperformed existing models, demonstrating superior accuracy in semi-supervised classification tasks. We have made our code publicly available at here to ensure the reproducibility of our results.

Multi-view local hyperplane nearest neighbor model based on independence criterion for identifying vesicular transport proteins

Vesicular transport proteins participate in various biological processes and play a significant role in the movement of substances within cells. These proteins are associated with numerous human diseases, making their identification particularly important. In this study, we developed a novel strategy for accurately identifying vesicular transport proteins. We developed a novel multi-view classifier called graph-regularized k-local hyperplane distance nearest neighbor model (HSIC-GHKNN), which combines the Hilbert–Schmidt independence criterion (HSIC)-based multi-view learning method with a local hyperplane distance nearest-neighbor classifier. We first extracted protein evolution information using two feature extraction methods, pseudo-position-specific scoring matrix (PsePSSM) and AATP, and addressed dataset imbalance using the Edited Nearest Neighbors (ENN) algorithm. Subsequently, we employed a local hyperplane distance nearest-neighbor classifier for each view identification and added an HSIC term to maintain independence between views. We then assessed the performance of our identification strategy and analyzed the PsePSSM and AATP feature sets to determine the influencing factors of the classification results. The experimental results demonstrate that the accurate and Matthew correlation coefficients of our strategy on the independent test set are 85.8 % and 0.548, respectively. Our approach outperformed existing methods in most evaluation metrics. In addition, the proposed multi-view classification model can easily be applied to similar identification tasks.

GraphLIME: Local Interpretable Model Explanations for Graph Neural Networks

Graph structured data has wide applicability in various domains such as physics, chemistry, biology, computer vision, and social networks, to name a few. Recently, graph neural networks (GNN) were shown to be successful in effectively representing graph structured data because of their good performance and generalization ability. However, explaining the effectiveness of GNN models is a challenging task because of the complex nonlinear transformations made over the iterations. In this paper, we propose GraphLIME, a local interpretable model explanation for graphs using the Hilbert-Schmidt Independence Criterion (HSIC) Lasso, which is a nonlinear feature selection method. GraphLIME is a generic GNN-model explanation framework that learns a nonlinear interpretable model locally in the subgraph of the node being explained. Through experiments on two real-world datasets, the explanations of GraphLIME are found to be of extraordinary degree and more descriptive in comparison to the existing explanation methods.

Differentially Private Nonlinear Causal Discovery from Numerical Data

Recently, several methods such as private ANM, EM-PC and Priv-PC have been proposed to perform differentially private causal discovery in various scenarios including bivariate, multivariate Gaussian and categorical cases. However, there is little effort on how to conduct private nonlinear causal discovery from numerical data. This work tries to challenge this problem. To this end, we propose a method to infer nonlinear causal relations from observed numerical data by using regression-based conditional independence test (RCIT) that consists of kernel ridge regression (KRR) and Hilbert-Schmidt independence criterion (HSIC) with permutation approximation. Sensitivity analysis for RCIT is given and a private constraint-based causal discovery framework with differential privacy guarantee is developed. Extensive simulations and real-world experiments for both conditional independence test and causal discovery are conducted, which show that our method is effective in handling nonlinear numerical cases and easy to implement. The source code of our method and data are available at https://github.com/Causality-Inference/PCD.

How is a global sensitivity analysis of a catchment-scale, distributed pesticide transfer model performed? Application to the PESHMELBA model

Abstract. Pesticide transfers in agricultural catchments are responsible for diffuse but major risks to water quality. Spatialized pesticide transfer models are useful tools to assess the impact of the structure of the landscape on water quality. Before considering using these tools in operational contexts, quantifying their uncertainties is a preliminary necessary step. In this study, we explored how global sensitivity analysis could be applied to the recent PESHMELBA pesticide transfer model to quantify uncertainties on transfer simulations. We set up a virtual catchment based on a real one, and we compared different approaches for sensitivity analysis that could handle the specificities of the model: a high number of input parameters and a limited size of sample due to computational cost and spatialized output. After a preliminary screening step, we calculated Sobol' indices obtained from polynomial chaos expansion, Hilbert–Schmidt independence criterion (HSIC) dependence measures and feature importance measures obtained from random forest surrogate model. Results from the different methods were compared regarding both the information they provide and their computational cost. Sensitivity indices were first computed for each landscape element (site sensitivity indices). Second, we proposed to aggregate them at the hillslope and the catchment scale in order to get a summary of the model sensitivity and a valuable insight into the model hydrodynamic behaviour. Conclusions about the advantages and disadvantages of each method may help modellers to conduct global sensitivity analysis on other such modular and distributed hydrological models as there has been a growing interest in these approaches in recent years.

Semantic Tradeoff for Heterogeneous Graph Embedding

Recently, many data mining models based on heterogeneous graph (HG) have emerged. Among these, HG embedding is an important and indispensable process. However, the existing HG embedding models usually use graph neural network to learn embeddings separately on different meta-paths, ignoring the fact that different meta-paths contain not only their unique semantics but also related semantics. This may result in semantic overlap or irrelevance, which needs to be a feasible and effective tradeoff for high-quality HG embedding, yet studies thereof have rarely been reported. In this article, we propose semantic tradeoff HG embedding (STHGE) by first introducing the Hilbert-Schmidt independence criterion (HSIC) as restriction. The main idea of STHGE is to regard semantic tradeoff as independence tradeoff (or correlation) between different meta-path spaces. Specifically, we first transform the original features of nodes into different meta-path feature spaces with HSIC restriction between them. Then, we use graph attention network to learn the embeddings of nodes on different meta-paths with HSIC restrictions. Finally, we concatenate the embeddings on different meta-paths to perform prediction. Experimental results on three heterogeneous datasets not only demonstrate the effectiveness of STHGE but also demonstrate that STHGE can achieve a new semantic tradeoff between different meta-paths. Furthermore, we demonstrate the robustness of STHGE.