Hilbert-Schmidt Independence Criterion Research Articles

Kernel-based Sensitivity Analysis for (Excursion) Sets

In this article, we aim to perform sensitivity analysis of set-valued models and, in particular, to quantify the impact of uncertain inputs on feasible sets, which are key elements in solving a robust optimization problem under constraints. While most sensitivity analysis methods deal with scalar outputs, this article introduces a novel approach to perform sensitivity analysis with set-valued outputs. Our innovative methodology is designed for excursion sets, but is versatile enough to be applied to set-valued simulators, including those found in viability fields, or when working with maps like pollutant concentration maps or flood zone maps. We propose to use the Hilbert-Schmidt Independence Criterion (HSIC) with a kernel designed for set-valued outputs. After setting a probabilistic framework for random sets, a first contribution is the proof that this kernel is characteristic, an essential property in a kernel-based sensitivity analysis context. To measure the contribution of each input, we then propose to use HSIC-ANOVA indices. With these indices, we can identify which inputs should be neglected (screening) and we can rank the others according to their influence (ranking). The estimation of these indices is also adapted to the set-valued outputs. Finally, we test the proposed method on three test cases of excursion sets.

A fast and accurate kernel-based independence test with applications to high-dimensional and functional data

Testing the dependency between two random variables is an important inference problem in statistics since many statistical procedures rely on the assumption that the two samples are independent. To test whether two samples are independent, a so-called HSIC (Hilbert–Schmidt Independence Criterion)-based test has been proposed. Its null distribution is approximated either by permutation or a Gamma approximation. In this paper, a new HSIC-based test is proposed. Its asymptotic null and alternative distributions are established. It is shown that the proposed test is root-n consistent. A three-cumulant matched chi-squared-approximation is adopted to approximate the null distribution of the test statistic. By choosing a proper reproducing kernel, the proposed test can be applied to many different types of data including multivariate, high-dimensional, and functional data. Three simulation studies and two real data applications show that in terms of level accuracy, power, and computational cost, the proposed test outperforms several existing tests for multivariate, high-dimensional, and functional data.

Community Detection in Multiplex Networks Using Orthogonal Non-Negative Matrix Tri-Factorization Based on Graph Regularization and Diversity

In recent years, community detection has received increasing interest. In network analysis, community detection refers to the identification of tightly connected subsets of nodes, which are called “communities” or “groups”, in the network. Non-negative matrix factorization models are often used to solve the problem. Orthogonal non-negative matrix tri-factorization (ONMTF) exhibits significant potential as an approach for community detection within multiplex networks. This paper explores the application of ONMTF in multiplex networks, aiming to detect both shared and exclusive communities simultaneously. The model decomposes each layer within the multiplex network into two low-rank matrices. One matrix corresponds to shared communities across all layers, and the other to unique communities within each layer. Additionally, graph regularization and the diversity of private communities are taken into account in the algorithm. The Hilbert Schmidt Independence Criterion (HSIC) is used to constrain the independence of private communities. The results prove that ONMTF effectively addresses community detection in multiplex networks. It also offers strong interpretability and feature extraction capabilities. Therefore, it is an advanced method for community detection in multiplex networks.

Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding

Deep neural networks are susceptible to catastrophic forgetting when trained on sequential tasks. Various continual learning (CL) methods often rely on exemplar buffers or/and network expansion for balancing model stability and plasticity, which, however, compromises their practical value due to privacy and memory concerns. Instead, this paper considers a strict yet realistic setting, where the training data from previous tasks is unavailable and the model size remains relatively constant during sequential training. To achieve such desiderata, we propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. This is achieved by the synergy between two key components: HSIC-Bottleneck Orthogonalization (HBO) implements non-overwritten parameter updates mediated by Hilbert-Schmidt independence criterion in an orthogonal space and EquiAngular Embedding (EAE) enhances decision boundary adaptation between old and new tasks with predefined basis vectors. Extensive experiments demonstrate that our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.

Prioritizing Causation in Decision Trees: A Framework for Interpretable Modeling

As a popular machine learning model, decision trees classify and generalize well, but face challenges in engineering applications: 1) Sensitivity to perturbations and lack of interpretability due to correlation reliance. 2) Manual setting of stopping criterion which is unrelated to correlation strength and easily leads to over-partitioning. To address these two challenges, we first theoretically analyze what leads to sub-optimal decision trees. By incorporating causal discovery, this limitation can be attributed to the fact that trees grown with spurious correlations often fall into sub-optimal that lead to overfitting and unfair behaviors. Neglecting causality motivates us to develop a ‘better’ tree with low Kolmogorov complexity and high generalization capability. Then we propose a causality decision tree framework, CausalDT, based on our theoretical expectation, where Hilbert-Schmidt independence criterion serves as a baseline. Unlike previous approaches that prioritize relevance, our framework determines branch nodes based on causation between features, with the significance level determining whether the tree should be expanded further. Experimental results demonstrate that our model maintains performance while reducing average tree depth by 35% on various datasets. Furthermore, our model enhances decision fairness and interpretability.

Hierarchical bottleneck for heterogeneous graph representation

Heterogeneous graphs (HGs) contain many nodes and their interaction relationships, which can model complex systems and provide rich semantic and structural information for task execution. Among these, HG representation stands as the fundamental and pivotal component. Existing HG representation methods primarily employ graph neural networks to acquire the semantics of nodes along various meta-paths and fuse them to represent the nodes. The most prevalent HG representation methods encompass two steps: semantic information extraction within meta-paths and semantic fusion between meta-paths. However, these methods overlooked the consideration of node heterogeneity within meta-paths and the simultaneous semantic correlation between meta-paths. Specifically, node heterogeneity within meta-paths signifies that the meta-path-based neighbors do not consistently contain information that positively influences the target node, and the semantic correlation between meta-paths indicates that different meta-path spaces are not entirely independent. Disregarding either of these issues leads to the propagation of irrelevant or redundant information and potential disruption of HG embedding. Consequently, in this study, we propose the HBHG, which is a hierarchical bottleneck for heterogeneous graph representation. HBHG primarily employs the information bottleneck (IB) as a guiding principle, constraining the propagation of irrelevant information within and between meta-paths while preserving relevant information. The central concept of the IB revolves around viewing model learning as the preservation of relevant information and compression of irrelevant information, accomplished by minimizing the dependency between input and hidden features through mutual information (MI) and maximizing the dependency between hidden features and ground-truth. Considering the complexity associated with MI estimation, this paper introduces a novel dependency index, namely the Hilbert-Schmidt independence criterion (HSIC), which offers ease of calculation. Specifically, HBHG comprises two primary components: a semantic bottleneck within meta-paths and a semantic bottleneck between meta-paths. The semantic bottleneck within meta-paths relies primarily on the HSIC-based limitations of dependencies at different layers of the graph neural network on various meta-paths, thereby maximizing the extraction of information relevant to the target node from neighboring nodes. The semantic bottleneck between meta-paths enables flexible extraction and fusion of semantic information based on downstream tasks, achieved by managing the trade-off of dependencies with HSIC between different meta-path semantic spaces. In summary, the proposed HBHG integrates hierarchical bottleneck constraints within and between meta-paths. This integration serves to maximize the aggregation of relevant information while effectively compressing irrelevant information, thereby enhancing the quality of heterogeneous graph embedding. The effectiveness of HBHG was validated through performance and ablation experiments conducted on multiple datasets.

Configuration Optimization of Temperature–Humidity Sensors Based on Weighted Hilbert–Schmidt Independence Criterion in Chinese Solar Greenhouses

For cost-sensitive Chinese solar greenhouses (CSGs) with an uneven spatial distribution in temperature and humidity, there is a lack of effective strategies for sensor configuration that can reduce sensor usage while monitoring the microclimate precisely. A configuration strategy for integrated temperature–humidity sensors (THSs) based on the improved weighted Hilbert–Schmidt independence criterion (HSIC) is proposed in this paper. The data independence of the THSs in different sites was analyzed based on the improved HSIC, and the selection priority of the THSs was ranked based on the weighted independence of temperature and humidity. Then, according to different cost constraints and monitoring requirements, suitable THSs could be selected sequentially and constitute the monitoring solution. Compared with the original monitoring solution containing twenty-two THSs, the optimized solution used only four THSs (S6, S9 and H6, H5) under strict cost constraints, with a maximum RMSE of the temperature and relative humidity of 0.6 °C and 2.30%, as well as a maximum information gain rate (IGR) of 9.47% and 10.0%. If higher monitoring precision is required, we can increase the THS usage with a greater budget. The optimized solution with six THSs (S6, S9, S8 and H6, H5, H2) could further reduce the maximum RMSE of the temperature and relative humidity to 0.33 °C and 1.10% and the IGR to 6.9% and 8.7%. This indicated that the proposed strategy could use much fewer THSs to achieve accurate and comprehensive monitoring, which would provide efficient and low-cost solutions for CSG microclimate monitoring.

ITTCA-MVL: A multi-view learning model based on physicochemical information and sequence statistical information for tumor T cell antigens identification

Immunotherapy is an emerging treatment method aimed at activating the human immune system and relying on its own immune function to kill cancer cells and tumor tissues. It has the advantages of wide applicability and minimal side effects. Effective identification of tumor T cell antigens (TTCAs) will help researchers understand their functions and mechanisms and carry out research on anti-tumor vaccine development. Considering that using biological experimental technology to identify TTCAs can be costly and time-consuming, it is necessary to develop a robust bioinformatics computing tool. At present, different machine learning models have been proposed for identifying TTCAs, but there is still room for further improvement in their performance. To establish a TTCA predictor with better prediction performance, we propose a prediction model called iTTCA-MVL in this paper. We extracted three sets of features from the views of physicochemical information and sequence statistics, namely the distribution descriptor of composition, transition, and distribution (CTDD), TF-IDF, and LSA topic. Then, we used least squares support vector machines (LSSVMs) as submodels and Hilbert‒Schmidt independence criteria (HSIC) as constraints to establish an independent and complementary multi-view learning model. The prediction accuracy of iTTCA-MVL on the independent test set is 0.873, and Matthew's correlation coefficient is 0.747, which is significantly better than those of existing methods. Therefore, iTTCA-MVL is an excellent prediction tool that researchers can use to accurately identify TTCAs.

Multiview Deep Subspace Clustering Networks.

Multiview subspace clustering aims to discover the inherent structure of data by fusing multiple views of complementary information. Most existing methods first extract multiple types of handcrafted features and then learn a joint affinity matrix for clustering. The disadvantage of this approach lies in two aspects: 1) multiview relations are not embedded into feature learning and 2) the end-to-end learning manner of deep learning is not suitable for multiview clustering. Even when deep features have been extracted, it is a nontrivial problem to choose a proper backbone for clustering on different datasets. To address these issues, we propose the multiview deep subspace clustering networks (MvDSCNs), which learns a multiview self-representation matrix in an end-to-end manner. The MvDSCN consists of two subnetworks, i.e., a diversity network (Dnet) and a universality network (Unet). A latent space is built using deep convolutional autoencoders, and a self-representation matrix is learned in the latent space using a fully connected layer. Dnet learns view-specific self-representation matrices, whereas Unet learns a common self-representation matrix for all views. To exploit the complementarity of multiview representations, the Hilbert-Schmidt independence criterion (HSIC) is introduced as a diversity regularizer that captures the nonlinear, high-order interview relations. Because different views share the same label space, the self-representation matrices of each view are aligned to the common one by universality regularization. The MvDSCN also unifies multiple backbones to boost clustering performance and avoid the need for model selection. Experiments demonstrate the superiority of the MvDSCN.

MORE POWERFUL HSIC-BASED INDEPENDENCE TESTS, EXTENSION TO SPACE-FILLING DESIGNS AND FUNCTIONAL DATA

The Hilbert-Schmidt independence criterion (HSIC) is a dependence measure based on reproducing kernel Hilbert spaces. This measure can be used for the global sensitivity analysis of numerical simulators whose objective is to identify the most influential inputs on the output(s) of the code. For this purpose, HSIC-based sensitivity measures and independence tests can be used for the ranking and screening of inputs, respectively. In this framework, this work proposes several improvements in the use of HSIC to increase their application spectrum and make the associated independence tests more powerful. First, we introduce a new method to perform the tests in a non-asymptotic framework. This method is much less central-processing-time expensive than the one based on permutation, while remaining as efficient. Then, the use of HSIC-based independence tests is extended to the case of some space-filling designs, where the independent and identically distributed condition of the observations is lifted. For this, a new procedure based on conditional randomization test is used. In addition, we also propose a more powerful test that relies on a well-chosen parameterization of the HSIC statistics: the kernel bandwidth parameter is optimized instead of the standard choices. Numerical studies are performed to assess the efficiency of these procedures and compare it to existing tests in the literature. Finally, HSIC-based indices for functional outputs are defined: they rely on appropriate and relevant kernels for this type of data. Illustrations are provided on temporal outputs of an analytical function and a compartmental epidemiological model.

Multi-label feature selection via constraint mapping space regularization

<abstract><p>Multi-label feature selection, an essential means of data dimension reduction in multi-label learning, has become one of the research hotspots in the field of machine learning. Because the linear assumption of sample space and label space is not suitable in most cases, many scholars use pseudo-label space. However, the use of pseudo-label space will increase the number of model variables and may lead to the loss of sample or label information. A multi-label feature selection scheme based on constraint mapping space regularization is proposed to solve this problem. The model first maps the sample space to the label space through the use of linear mapping. Second, given that the sample cannot be perfectly mapped to the label space, the mapping space should be closest to the label space and still retain the space of the basic manifold structure of the sample space, so combining the Hilbert-Schmidt independence criterion with the sample manifold, basic properties of constraint mapping space. Finally, the proposed algorithm is compared with MRDM, SSFS, and other algorithms on multiple classical multi-label data sets; the results show that the proposed algorithm is effective on multiple indicators.</p></abstract>

Stacked maximum independence autoencoders: A domain generalization approach for fault diagnosis under various working conditions

One of the major obstacles in rotating machinery fault diagnosis is the distribution discrepancy in feature space caused by the change of working conditions. To solve this problem, some single or multiple source domain adaptation methods have been developed. They minimize the distribution discrepancy between a source domain (or several source domains) and a target domain by maximum mean discrepancy (MMD) or its variants. However, the application scope of these methods is limited to the target domain used in adaptation training. The trained adaptation models are not generally adapted thus cannot be directly used for other unseen new domains. To address this limitation, this study proposes a generally domain adaptable approach-maximum independence stacked autoencoders (MI-SAE). First, a domain label which describes the working condition of a sample is defined. Then, maximum independence autoencoder (MI-AE) is proposed to minimize the dependence between the extracted features and the corresponding domain labels using Hilbert-Schmidt Independence Criterion (HSIC) instead of MMD. By stacking multiple MI-AEs and finetune them with labeled source samples and normal pattern samples of other supporting domains, MI-SAE is obtained. The proposed approach can learn generally domain invariant features which has maximum independence with working conditions by HSIC,thus overcome the limitations of MMD based domain adaptations. The proposed approach is applied to diagnose the faults of gearbox and engine rolling bearing. Results show it outperforms state-of-the-art single or multiple source domain adaptation models and has more practicability due to its lower requirement for labeled data.

One step at a time: A statistical approach for distinguishing mediators, confounders, and colliders using direction dependence analysis.

In observational data, understanding the causal link when estimating the causal effect of an independent variable (x) on a dependent variable (y) often requires researchers to identify the role of a third variable in the x → y relationship. Mediation, confounding, and colliding are three key third-variable effects that yield different theoretical and methodological implications for drawing causal conclusions. Commonly used covariance-based statistical methods, such as linear regression and structural equation modeling, cannot distinguish these effects in practice, however. In this study, we introduce a statistical approach for distinguishing mediators, confounders, colliders, and potential M-bias structures that uses higher-order moment information from the data. We propose a two-step procedure that uses the Hilbert-Schmidt independence criterion within the direction dependence analysis framework. Results from Monte Carlo simulations show that our proposed approach accurately recovers the true data-generating process of the third variable. We provide an empirical example to demonstrate the application of our proposed approach in psychological research. Finally, we discuss implications and future directions of our work. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Testing independence of functional variables by an Hilbert–Schmidt independence criterion estimator

We propose a method for testing independence of functional variables by using an estimator of the Hilbert–Schmidt Independence Criterion obtained from an appropriate modification of the usual estimator. We get asymptotic normality of this estimator both under independence hypothesis and under the alternative hypothesis. A simulation study that allows to compare the proposed test to an existing one is provided.

Consistency–exclusivity guided unsupervised multi-view feature selection

Unsupervised multi-view feature selection (UMFS) is an effective dimension reduction for multi-view data. It aims to obtain the important feature subset from multi-view data, which can significantly minimize the impacts of noises, outliers and redundancy. Although previous UMFS methods achieve remarkable achievements, they fail to fully take into account the consistency or the exclusivity hidden in multi-view data. To address this, this article presents a consistency–exclusivity guided unsupervised multi-view feature selection (CE-UMFS) method. Specifically, we design a multi-view matrix factorization model to simultaneously explore the consistency and exclusivity in multi-view data. Meanwhile, we employ the Hilbert–Schmidt independence criterion (HSIC) to preserve the exclusivity of different views. Furthermore, we impose a nuclear norm on the consistent representation matrix to explore the consistency across views. At last, promising experimental results demonstrate the superiority of the proposed method compared with some state-of-the-art methods.

Multi-target HSIC-Lasso via exploiting target correlations

Multi-target regression (MTR) has been widely applied due to its compact structure and better computational efficiency in learning multiple related tasks, and has received extensive attention in machine learning, while great challenge arises from handling complex nonlinear relationships in high-dimensional data. In this study, we propose a novel approach to nonlinear dimensionality reduction in MTR. Specifically, the Hilbert–Schmidt Independence Criterion Lasso (HSIC-Lasso) is introduced to captures nonlinear dependencies between input–output and inter-target relationships in a unified framework. Additionally, sparse learning is incorporated to effectively eliminate redundant and irrelevant features. The proposed method is guaranteed to find an optimal subset of maximally predictive features with minimal redundancy, and explores target correlations by treating the targets as input features, which achieves a better learning of the target dependencies and thus improves the effectiveness of the feature selection. Extensive experiments on twelve public data sets have shown that the proposed methods outperform the compared methods in feature selection for the MTR problem.

Optimizing Graph Neural Network With Multiaspect Hilbert-Schmidt Independence Criterion.

The graph neural network (GNN) has demonstrated its superior power in various data mining tasks and has been widely applied in diversified fields. The core of GNN is the aggregation and combination functions, and mainstream GNN studies focus on the enhancement of these functions. However, GNNs face a common challenge, i.e., useless features contained in neighbor nodes may be integrated into the target node during the aggregation process. This leads to poor node embedding and undermines downstream tasks. To tackle this problem, this article proposes a novel GNN optimization framework GNN-MHSIC by introducing the nonparametric dependence method Hilbert-Schmidt independence criterion (HSIC) under the guidance of information bottleneck. HSIC is utilized to guide the information propagation among layers of a GNN from multiaspect views. GNN-MHSIC aims to achieve three main objectives: 1) minimizing the HSIC between the input features and the propagation layers; 2) maximizing the HSIC between the propagation layers and the ground truth; and 3) minimizing the HSIC between the propagation layers. With a multiaspect design, GNN-MHSIC can minimize the propagation of redundant information while preserving relevant information about the target node. We prove GNN-MHSIC's finite upper and lower bounds theoretically and evaluate it experimentally with four classic GNN models, including the graph convolutional network, the graph attention network (GAT), the heterogeneous GAT, and the heterogeneous graph (HG) propagation network on three widely used HGs. The results illustrate the usefulness and performance of GNN-MHSIC.

Discriminative multi-label feature selection with adaptive graph diffusion

Feature selection can alleviate the problem of the curse of dimensionality by selecting more discriminative features, which plays an important role in multi-label learning. Recently, embedded feature selection methods have received increasing attentions. However, most existing methods learn the low-dimensional embeddings under the guidance of the local structure between the original instance pairs, thereby ignoring the high-order structure between instances and being sensitive to noise in the original features. To address these issues, we propose a feature selection method named discriminative multi-label feature selection with adaptive graph diffusion (MFS-AGD). Specifically, we first construct a graph embedding learning framework equipped with adaptive graph diffusion to uncover a latent subspace that preserves the higher-order structure information between four tuples. Then, the Hilbert–Schmidt independence criterion (HSIC) is incorporated into the embedding learning framework to ensure the maximum dependency between the latent representation and labels. Benefiting from the interactive optimization of the feature selection matrix, latent representation and similarity graph, the selected features can accurately explore the higher-order structural and supervised information of data. By further considering the correlation between labels, MFS-AG is extended to a more discriminative version,i.e., LMFS-AG. Extensive experimental results on various benchmark data sets validate the advantages of the proposed MFS-AGD and LMFS-AGD methods.

Kernel-Based Independence Tests for Causal Structure Learning on Functional Data.

Measurements of systems taken along a continuous functional dimension, such as time or space, are ubiquitous in many fields, from the physical and biological sciences to economics and engineering. Such measurements can be viewed as realisations of an underlying smooth process sampled over the continuum. However, traditional methods for independence testing and causal learning are not directly applicable to such data, as they do not take into account the dependence along the functional dimension. By using specifically designed kernels, we introduce statistical tests for bivariate, joint, and conditional independence for functional variables. Our method not only extends the applicability to functional data of the Hilbert-Schmidt independence criterion (hsic) and its d-variate version (d-hsic), but also allows us to introduce a test for conditional independence by defining a novel statistic for the conditional permutation test (cpt) based on the Hilbert-Schmidt conditional independence criterion (hscic), with optimised regularisation strength estimated through an evaluation rejection rate. Our empirical results of the size and power of these tests on synthetic functional data show good performance, and we then exemplify their application to several constraint- and regression-based causal structure learning problems, including both synthetic examples and real socioeconomic data.

Kernel-based joint independence tests for multivariate stationary and non-stationary time series.

Multivariate time-series data that capture the temporal evolution of interconnected systems are ubiquitous in diverse areas. Understanding the complex relationships and potential dependencies among co-observed variables is crucial for the accurate statistical modelling and analysis of such systems. Here, we introduce kernel-based statistical tests of joint independence in multivariate time series by extending the d-variable Hilbert-Schmidt independence criterion to encompass both stationary and non-stationary processes, thus allowing broader real-world applications. By leveraging resampling techniques tailored for both single- and multiple-realization time series, we show how the method robustly uncovers significant higher-order dependencies in synthetic examples, including frequency mixing data and logic gates, as well as real-world climate, neuroscience and socio-economic data. Our method adds to the mathematical toolbox for the analysis of multivariate time series and can aid in uncovering high-order interactions in data.