Hilbert-Schmidt Independence Criterion Research Articles

Array Optimization Based on Weighted and Hilbert–Schmidt Schemes of Multisensor Detection System

This paper presents a novel sensor array optimization scheme for multisensor electronic nose detection system. A system architecture with multisensor is first proposed to implement the medical detection, including the bacterial culture medium detection and animal wound infection detection. The system efficiency is evaluated by comparing with the field asymmetric ion mobility spectrometry (FAIMS) system. To further improve the detection effect and reduce the number of sensors of the electronic nose system, we then derive two sensor array optimization procedures based on factor analysis and Hilbert-Schmidt independence criterion, respectively. Specifically, the weighted factor analysis method and non-weighted factor analysis method are proposed via factor analysis. Besides, the Hilbert-Schmidt independence criterion optimization design of linear kernel function and Gaussian kernel function are also exploited. The experiment results highlight that compared with the existing approaches, the proposed weighted factor analysis optimization method and Hilbert-Schmidt independence criterion optimization method (Gaussian kernel function) can achieve a significant system performance.

Gaussian RBF Centered Kernel Alignment (CKA) in the Large-Bandwidth Limit.

Centered kernel alignment (CKA), also known as centered kernel-target alignment, is useful as a similarity measure between kernels and as a kernel-based similarity measure between feature representations. We prove that CKA based on a Gaussian RBF kernel converges to linear CKA in the large-bandwidth limit. The result relies on mean-centering of the feature maps and on a Hilbert-Schmidt Independence Criterion (HSIC) identity. We show that convergence onset is sensitive to the geometry of the feature representations, and that a notion of representation eccentricity, ρ, constrains the bandwidth range for which Gaussian CKA can differ noticeably from linear CKA. Our experimental results suggest that Gaussian bandwidths less than ρ should be selected in order to enable nonlinear modeling.

Sensitivity Analysis for ReaxFF Reparametrization Using the Hilbert-Schmidt Independence Criterion.

We apply a global sensitivity method, the Hilbert-Schmidt independence criterion (HSIC), to the reparametrization of a Zn/S/H ReaxFF force field to identify the most appropriate parameters for reparametrization. Parameter selection remains a challenge in this context, as high-dimensional optimizations are prone to overfitting and take a long time but selecting too few parameters leads to poor-quality force fields. We show that the HSIC correctly and quickly identifies the most sensitive parameters and that optimizations done using a small number of sensitive parameters outperform those done using a higher-dimensional reasonable-user parameter selection. Optimizations using only sensitive parameters (1) converge faster, (2) have loss values comparable to those found with the naive selection, (3) have similar accuracy in validation tests, and (4) do not suffer from problems of overfitting. We demonstrate that an HSIC global sensitivity is a cheap optimization preprocessing step that has both qualitative and quantitative benefits which can substantially simplify and speed up ReaxFF reparametrizations.

A unified view of feature selection based on Hilbert-Schmidt independence criterion

Feature selection is a challenging and increasing important task in the machine learning and data mining community. According to different learning scenarios such as the availability of supervision information, feature selection can be broadly categorized as supervised, unsupervised and semi-supervised methods. Traditionally, feature selection algorithms are scenario-specific, i.e., different types of feature selection algorithms were proposed and investigated separately for specific learning scenario. In this paper, we aim to develop a unified view of supervised, unsupervised and semi-supervised feature selection methods. Based on the Hilbert-Schmidt independence criterion (HSIC), we first show that supervised, unsupervised and semi-supervised feature selection methods share the same objective. In discussing the connection between the unified framework and HSIC least absolute shrinkage and selection operator (HSIC Lasso), we find that the proposed framework not only has a clear statistical interpretation that minimum redundant features with maximum dependence on the target output variable can be found in terms of the HSIC, but also enables the global optimal solution to be computed efficiently by solving a Lasso optimization problem. Under this unified view, a new unsupervised feature selection algorithm is also proposed and demonstrated with several benchmark examples.

A novel hybrid variable selection strategy with application to molecular spectroscopic analysis

In chemometrics, partial least squares (PLS) regression has become an established tool for spectroscopic analysis. Even so, to improve multivariate calibration models’ performance, variable selection is necessary for molecular spectroscopic analysis in many scenarios. However, applying only one variable selection method has many limitations. For example, individual wavelength selection methods may suffer from large computational requirements. In addition, wavelength intervals selection methods may suffer from the retention of uninformative/interfering wavelengths and the effect of collinearity in the selected wavelength intervals. Accordingly, a novel hybrid variable selection strategy, called memetic algorithm-interval partial least squares coupled Hilbert-Schmidt independence criterion based variable space iterative optimization (MA-iPLS + HSIC–VSIO), is proposed in this study. In the first step, a wavelength intervals selection method, MA-iPLS, is used to select optimal wavelength intervals. In the second step, a novel individual wavelength selection method, HSIC–VSIO, is employed for further optimizing the selected wavelength intervals. This hybrid variable selection strategy makes full use of MA-iPLS and HSIC–VSIO. To investigate the performance of MA-iPLS + HSIC–VSIO, it was tested on two groups of spectroscopic datasets: the surface-enhanced Raman scattering (SERS) spectra of chlorpyrifos standard solutions dataset and the near infrared (NIR) spectra of diesel fuels dataset. Nine methods, including PLS, HSIC–VISO, MA-iPLS, genetic algorithm-interval partial least squares (GA-iPLS), particle swarm optimization-interval partial least squares (PSO-iPLS), interval random frog (iRF), GA-iPLS + HSIC–VSIO, PSO-iPLS + HSIC–VSIO, and iRF + HSIC–VSIO, were also applied on the spectroscopic dataset for comparison. The results demonstrated the excellent performance of MA-iPLS + HSIC–VSIO for molecular spectroscopic analysis.

Extending Hilbert-Schmidt Independence Criterion for Testing Conditional Independence.

The Conditional Independence (CI) test is a fundamental problem in statistics. Many nonparametric CI tests have been developed, but a common challenge exists: the current methods perform poorly with a high-dimensional conditioning set. In this paper, we considered a nonparametric CI test using a kernel-based test statistic, which can be viewed as an extension of the Hilbert-Schmidt Independence Criterion (HSIC). We propose a local bootstrap method to generate samples from the null distribution H0:X⫫Y∣Z. The experimental results showed that our proposed method led to a significant performance improvement compared with previous methods. In particular, our method performed well against the growth of the dimension of the conditioning set. Meanwhile, our method can be computed efficiently against the growth of the sample size and the dimension of the conditioning set.

Multiple color representation and fusion for diabetes mellitus diagnosis based on back tongue images

Tongue images have been proved to be effective in diabetes mellitus (DM) diagnosis. Without requirement of collecting blood sample, tongue image based diagnosis approach is non-invasive and convenient for the patients. Meanwhile, the colors of tongues play an important in aiding accurate diagnosis. However, the tongues’ colors fall on a small color gamut that makes it difficult for the existing color descripts to identify and distinguish the tiny difference of the tongues. To tackle this problem, we introduce a novel color descriptor by representing the colors with the clustering centers, namely color centroid points, of the color points sampled from tongue images. In order to boost the capacity of the descriptor, we extend it into three color spaces, i.e., RGB, HSV and LAB to mine a rich set of color information and exploit the complementary information among the three spaces. Since there exist correlation and complementarity among the features extracted from the three color spaces, we propose a novel multiple color features fusion method for DM diagnosis. Particularly, two projections are learned to project the multiple features to their corresponding shared and specific subspaces, in which their similarity and diversity are firstly measured by the Euclidean Distance and Hilbert Schmidt Independence Criterion (HSIC), respectively. To fully exploit the similar and complementary information, the two components are jointly transformed to their label vector, efficiently embedding the discriminant prior into the model, leading to significant improvement in the diagnosis outcomes. Experimental results on clinical tongue dataset substantiated the effectiveness of our proposed clustering-based color descriptor and the proposed multiple colors fusion approach. Overall, the proposed pipeline for the diagnosis of DM using back tongue images, achieved an average accuracy of up to 93.38%, indicating its potential toward realization of a clinical diagnostic tool for DM. Without loss generality, we also assessed the performance of the novel multiple features fusion method on two public datasets. The experiments prove the superiority of our multiple features learning model on general real-life application.

Independence tests with random subspace of two random vectors in high dimension

Testing for independence between two random vectors is a fundamental problem in statistics. When the dimension of these two random vectors are fixed, the existing tests based on the distance covariance and Hilbert–Schmidt independence criterion with many desirable properties, including the capacity to capture linear and non-linear dependence. However, these tests may fail to capture the non-linear dependence due to the “curse of dimensionality” when the random vectors are high dimensional. To attack this problem, we propose a general framework for testing the dependence of two random vectors to randomly select two subspaces consisting of components of the vectors, respectively. To enhance the performance of this method, we repeatedly implement this procedure to construct the final test statistic. The new method can also work for non-linear dependence detection in a high-dimensional setup. Theoretically, if the replication time tends to infinity to get the final statistic, we can avoid potential power loss caused by lousy subspaces. Therefore, the two proposed tests are consistent with general alternatives. The weak limit under the null hypothesis is normal; thus, determining critical value need not resort to resampling approximation. We demonstrate the finite-sample performance of the proposed test by using Monte Carlo simulations and the analysis for a real-data example.

Data-Driven Modeling Through the Moodle Learning Management System: An Empirical Study Based on a Mathematics Teaching Subject

This work addresses the problem of inferring student performance from information acquired in a Learning Management System (LMS). In particular, we explore the capabilities offered by Moodle, a widely used LMS. The study is performed on data acquired from four classes within the same subject, for whom we make inferences about student performance marks. The developed methodology describes the degree in which the acquired information allows to predict the student marks belonging to the continuous evaluation, while the prediction of the final students marks has a higher intrinsic complexity that would require a more in-depth study of the relevant variables. Then, we follow a fully data-driven process to discover similarities among classes. In particular, we propose the use of a dependence estimation measure, the normalized Hilbert-Schmidt Independence criterion. We show how this dependence measure is useful to determine relations among classes, based on their particular teaching methodology, by only using data acquired from the LMS. This opens the door to explore the capabilities of LMS in similarity search between offered courses along an educational platform. In order to help the community and serve as a common way of comparison, we provide the source code of the proposed methodology.

Goal-Oriented Sensitivity Analysis of Hyperparameters in Deep Learning

Tackling new machine learning problems with neural networks always means optimizing numerous hyperparameters that define their structure and strongly impact their performances. In this work, we study the use of goal-oriented sensitivity analysis, based on the Hilbert–Schmidt independence criterion (HSIC), for hyperparameter analysis and optimization. Hyperparameters live in spaces that are often complex and awkward. They can be of different natures (categorical, discrete, boolean, continuous), interact, and have inter-dependencies. All this makes it non-trivial to perform classical sensitivity analysis. We alleviate these difficulties to obtain a robust analysis index that is able to quantify hyperparameters’ relative impact on a neural network’s final error. This valuable tool allows us to better understand hyperparameters and to make hyperparameter optimization more interpretable. We illustrate the benefits of this knowledge in the context of hyperparameter optimization and derive an HSIC-based optimization algorithm that we apply on MNIST and Cifar, classical machine learning data sets, but also on the approximation of Runge function and Bateman equations solution, of interest for scientific machine learning. This method yields neural networks that are both competitive and cost-effective.

Multi-label feature selection based on HSIC and sparrow search algorithm.

Feature selection has always been an important topic in machine learning and data mining. In multi-label learning tasks, each sample in the dataset is associated with multiple labels, and labels are usually related to each other. At the same time, multi-label learning has the problem of "curse of dimensionality". Feature selection therefore becomes a difficult task. To solve this problem, this paper proposes a multi-label feature selection method based on the Hilbert-Schmidt independence criterion (HSIC) and sparrow search algorithm (SSA). It uses SSA for feature search and HSIC as feature selection criterion to describe the dependence between features and all labels, so as to select the optimal feature subset. Experimental results demonstrate the effectiveness of the proposed method.

Learning Spatial Regularization Correlation Filters With the Hilbert-Schmidt Independence Criterion in RKHS for UAV Tracking

Unmanned aerial vehicle (UAV) object tracking have been widely employed in military and civilian tasks and is very challenging due to factors such as aerial view, low resolution, background clutter, and small scale. Currently, many correlation filter (CF) methods for UAV object tracking introduce predefined regularization terms to improve the discriminative power. However, the predefined spatial constraints neither exploit the changing foreground and background information during tracking nor adapt to complex video attributes, such as background clutter, illumination variation, and viewpoint change. In this work, a novel CF tracking model is proposed to learn the spatial regularization correlation filters with the Hilbert-Schmidt independence criterion (HSIC_SRCF) in the reproducing kernel Hilbert space (RKHS) for UAV tracking. First, HSIC_SRCF learns the high confidence spatial weight coefficient for the filters, which encourages the filters to focus on the more reliable region. Second, the alternative direction method of multipliers (ADMM) can be used to optimize the proposed model, and the subproblems of the model have closed-form solutions. Third, the model update scheme uses RKHS and the peak-to-sidelobe ratio to obtain the confidence of the object bounding box and reduce the effect of background noise for the UAV object. The ablation study shows that the proposed approach achieves a precision of 0.699 and a success rate of 0.607, which are 2.8% higher and 2.2% higher, respectively, than the baseline AutoTrack on the UAV123@10fps. Experimental results on four popular UAV datasets, including VisDrone2018-test-dev, UAV123@10fps, UAV123, and UAVDT, show that our approach performs favorably compared with current state-of-the-art methods.

Consistency and Diversity Induced Human Motion Segmentation.

Subspace clustering is a classical technique that has been widely used for human motion segmentation and other related tasks. However, existing segmentation methods often cluster data without guidance from prior knowledge, resulting in unsatisfactory segmentation results. To this end, we propose a novel Consistency and Diversity induced human Motion Segmentation (CDMS) algorithm. Specifically, our model factorizes the source and target data into distinct multi-layer feature spaces, in which transfer subspace learning is conducted on different layers to capture multi-level information. A multi-mutual consistency learning strategy is carried out to reduce the domain gap between the source and target data. In this way, the domain-specific knowledge and domain-invariant properties can be explored simultaneously. Besides, a novel constraint based on the Hilbert Schmidt Independence Criterion (HSIC) is introduced to ensure the diversity of multi-level subspace representations, which enables the complementarity of multi-level representations to be explored to boost the transfer learning performance. Moreover, to preserve the temporal correlations, an enhanced graph regularizer is imposed on the learned representation coefficients and the multi-level representations of the source data. The proposed model can be efficiently solved using the Alternating Direction Method of Multipliers (ADMM) algorithm. Extensive experimental results on public human motion datasets demonstrate the effectiveness of our method against several state-of-the-art approaches.

Diversity Multi-View Clustering With Subspace and NMF-Based Manifold Learning

Since the complementarity information among multiple views has been exploited to improve the clustering effect significantly, multi-view clustering has become a hot topic, and many multi-view clustering methods have emerged. Most of them only consider local features in each view, ignoring the differences in the manifold structure of the same class samples among different views. In addition, they need to balance the importance of respective views effectively, thus ignoring the diversity among views in the clustering process. To address these problems, we propose a new diversity multi- view clustering method with subspace and NMF-based manifold learning. Firstly, non-negative matrix factorization is utilized to obtain the samples’ features and local geometric structure. After that, manifold learning and latent representation facilitate the learning of standard geometric structures in different views. Moreover, the Hilbert-Schmidt independence criterion is introduced to learn diversity for mutual learning and information fusion among views. Finally, experiments on seven datasets demonstrate the superiority of the proposed method compared to ten state-of-the-art methods.

Towards Adaptive Consensus Graph: Multi-View Clustering via Graph Collaboration

Multi-view clustering is a long-standing important task, however, it remains challenging to exploit valuable information from the complex multi-view data located in diverse highdimensional spaces. The core issue is the effective collaboration of multiple views to holistically uncover the essential correlations between multi-view data through graph learning. Furthermore, it is indispensable for most existing methods to introduce an additional clustering step to produce the final clusters, which evidently reduces the uniform relationship between graph learning and clustering. Based on the above considerations, in this paper, we present a novel method named multi-view clustering via graph collaboration (MCGC). Based on the low-dimensional representation space developed by MCGC, it first perceives the correlations between samples in each individual view under the supervision of the Hilbert-Schmidt independence criterion (HSIC). Then, MCGC proposes learning a consensus graph by adaptively collaborating between all the views, which is able to uncover the essential structure of the multi-view data. Meanwhile, by imposing the rank constraint on the Laplacian matrix of the consensus graph to partition the multi-view data naturally into the required number of clusters, the optimal clustering results can be obtained directly without any postprocessing steps. Finally, the resulting optimization problem is solved by an alternating optimization scheme with guaranteed fast convergence. Extensive experiments on 5 benchmark multi-view datasets demonstrate that MCGC markedly outperforms the state-of-the-art baselines.

Multivariate tests of independence based on a new class of measures of independence in Reproducing Kernel Hilbert Space

In this paper, we propose a new class of independence measures based on the maximum mean discrepancy (MMD) in Reproducing Kernel Hilbert Space (RKHS). We use a novel way to build the independence measure by creating a metric on the space of all Borel probability measures. The proposed approach has several attractive properties including (i) no required assumptions about the data structure; (ii) insensitive to the dimension of the data; (iii) being more flexible than the Hilbert–Schmidt independence criterion (HSIC), which is the most popular independence measure based on RKHS. We show that the empirical estimator of the proposed independence measure possesses some desirable large sample properties regardless of the dimension of data. Based on the proposed independence measure, we develop two tests of independence in which the test statistics have simple forms and are easy to compute. The performance of the proposed tests of independence for high-dimensional data is evaluated through an extensive Monte Carlo simulation study.

A fast kernel independence test for cluster-correlated data

Cluster-correlated data receives a lot of attention in biomedical and longitudinal studies and it is of interest to assess the generalized dependence between two multivariate variables under the cluster-correlated structure. The Hilbert–Schmidt independence criterion (HSIC) is a powerful kernel-based test statistic that captures various dependence between two random vectors and can be applied to an arbitrary non-Euclidean domain. However, the existing HSIC is not directly applicable to cluster-correlated data. Therefore, we propose a HSIC-based test of independence for cluster-correlated data. The new test statistic combines kernel information so that the dependence structure in each cluster is fully considered and exhibits good performance under high dimensions. Moreover, a rapid p value approximation makes the new test fast applicable to large datasets. Numerical studies show that the new approach performs well in both synthetic and real world data.

Geometry-Guided Dense Perspective Network for Speech-Driven Facial Animation.

Realistic speech-driven 3D facial animation is a challenging problem due to the complex relationship between speech and face. In this paper, we propose a deep architecture, called Geometry-guided Dense Perspective Network (GDPnet), to achieve speaker-independent realistic 3D facial animation. The encoder is designed with dense connections to strengthen feature propagation and encourage the re-use of audio features, and the decoder is integrated with an attention mechanism to adaptively recalibrate point-wise feature responses by explicitly modeling interdependencies between different neuron units. We also introduce a non-linear face reconstruction representation as a guidance of latent space to obtain more accurate deformation, which helps solve the geometry-related deformation and is good for generalization across subjects. Huber and HSIC (Hilbert-Schmidt Independence Criterion) constraints are adopted to promote the robustness of our model and to better exploit the non-linear and high-order correlations. Experimental results on the public dataset and real scanned dataset validate the superiority of our proposed GDPnet compared with state-of-the-art model. The code is available for research purposes at http://cic.tju.edu.cn/faculty/likun/projects/GDPnet.

A Multi-View Deep Metric Learning approach for Categorical Representation on mixed data

It is an important and challenging task to represent the categorical values in mixed data as numerical vectors with intrinsic features, by revealing the complex coupling relationships between the categorical values, attributes and samples. The majority of extant studies expose only one particular coupling relationship in depth, or fuse multiple coupling relationships by using shallow learning based on kernels. The former may not fully mine the essential features of the categorical data. The latter typically has some limitations, for example, difficulty in expanding the spatial structure and difficulty in determining the optimal kernel function. Therefore, this paper proposes a Multi-view Deep Metric Learning for Categorical Representation on mixed data (MvDML-CR). Specifically, first, based on the principle of information complementarity, multiple coupled views are extracted from the complex interaction relationships of the categorical data. Then, in each coupled view, a new proxy loss function is designed to build a deep metric learning sub-model with strong separability, which represents the categorical values as numerical vectors with discrimination. Last, we employ the Hilbert–Schmidt independence criterion to maximize the dependency between the views, and then fuse the sub-models trained in the different views to enhance the complementarity and consistency of the categorical representations. Extensive experiments on 34 mixed datasets with diversified characteristics demonstrate that the classification performance of MvDML-CR is significantly improved, compared with the state-of-the-art competitors.

Robust multi-view subspace enhanced representation based on collaborative constraints and HSIC induction

The design of effective multi-view subspace clustering (MSC) algorithms has recently garnered significant research attention. Herein, to effectively improve the recognition performance and anti-noise interference ability of an MSC model, we propose a novel MSC algorithm, termed as robust multi-view subspace enhancement representation, based on collaborative constraints and a Hilbert–Schmidt independence criterion (HSIC) induction method. To mine the complementary information between different views, we apply the HSIC as a diversity regularization term. Specifically, to enhance the diagonal block structure of a subspace representation, a new sparse constraint is introduced on the product of itself and the transpose of the subspace representation matrix in a multi-view subspace learning model. Furthermore, hypergraph regularization and a low-rank idea are considered to capture the local geometric structure and clean data. In addition, to optimize our model, we adopt an augmented Lagrangian multiplier method and discuss the convergence of the model. Extensive experiments on six challenging datasets reveal that the proposed method achieves a highly competent objective performance with and without noisy views, as compared with several state-of-the-art multi-view clustering methods.