Statistical Guarantees Research Articles

Distributed Estimation of Principal Support Vector Machines for Sufficient Dimension Reduction

The principal support vector machines method (Li et al., 2011) is a powerful tool for sufficient dimension reduction that replaces original predictors with their low-dimensional linear combinations while preserving the information for regression and classification. However, the computational burden of the principal support vector machines method constrains its use for massive data. To address this issue, we propose a naive and a refined distributed estimation algorithms for fast implementation when the sample size is large. Both distributed sufficient dimension reduction estimators exhibit the same statistical efficiency as when all the data is merged together, which provides rigorous statistical guarantees for their application to large-scale datasets, while the refined method requires smaller batch sample sizes and hence is more advantageous when memory limitations exist on distributed machines. The two distributed algorithms are further adapted to principal weighted support vector machines (Shin et al., 2017) for sufficient dimension reduction in binary classification. The statistical accuracy and computational complexity of our proposed methods are examined through comprehensive simulation studies and in a real data application with more than 600000 samples.

Heterogeneous functional regression for subgroup analysis

With ever increasing number of features of modern datasets, data heterogeneity is gradually becoming the norm rather than the exception. Whereas classical regressions usually assume all the samples follow a common model, it becomes imperative to identify the heterogeneous relationship in different subsamples. In this article, we propose a new approach to model heterogeneous functional regression relations. We target at the association between a response and a predictor, whose relationship can vary across underlying subgroups and is modeled as an unknown functional of an auxiliary predictor. We introduce a procedure which performs simultaneous parameter estimation and subgroup identification through a fusion type group-wise penalization. We establish the statistical guarantees in terms of non-asymptotic convergence of the parameter estimation. We also establish the oracle property and asymptotic normality of the estimators. We carry out intensive simulations, and illustrate with a new dataset from an Alzheimer’s disease study.

Dynamic frequent subgraph mining algorithms over evolving graphs: a survey

Frequent subgraph mining (FSM) is an essential and challenging graph mining task used in several applications of the modern data science. Some of the FSM algorithms have the objective of finding all frequent subgraphs whereas some of the algorithms focus on discovering frequent subgraphs approximately. On the other hand, modern applications employ evolving graphs where the increments are small graphs or stream of nodes and edges. In such cases, FSM task becomes more challenging due to growing data size and complexity of the base algorithms. Recently we see frequent subgraph mining algorithms designed for dynamic graph data. However, there is no comparative review of the dynamic subgraph mining algorithms focusing on the discovery of frequent subgraphs over evolving graph data. This article focuses on the characteristics of dynamic frequent subgraph mining algorithms over evolving graphs. We first introduce and compare dynamic frequent subgraph mining algorithms; trying to highlight their attributes as increment type, graph type, graph representation, internal data structure, algorithmic approach, programming approach, base algorithm and output type. Secondly, we introduce and compare the approximate frequent subgraph mining algorithms for dynamic graphs with additional attributes as their sampling strategy, data in the sample, statistical guarantees on the sample and their main objective. Finally, we highlight research opportunities in this specific domain from our perspective. Overall, we aim to introduce the research area of frequent subgraph mining over evolving graphs with the hope that this can serve as a reference and inspiration for the researchers of the field.

Statistical Testing of Quantum Programs via Fixed-Point Amplitude Amplification

We present a new technique for accelerating quantum program testing. Given a quantum circuit with an input/output specification, our goal is to check whether executing the program on the input state produces the expected output. In quantum computing, however, it is impossible to directly check the equivalence of the two quantum states. Instead, we rely on statistical testing, which involves repeated program executions, state measurements, and subsequent comparisons with the specified output. To guarantee a high level of assurance, however, this method requires an extensive number of measurements. In this paper, we propose a solution to alleviate this challenge by adapting Fixed-Point Amplitude Amplification (FPAA) for quantum program testing. We formally present our technique, demonstrate its ability to reduce the required number of measurements as well as runtime cost without sacrificing the original statistical guarantee, and showcase its runtime effectiveness through case studies.

Net-Zero Scheduling of Multi-Energy Building Energy Systems: A Learning-Based Robust Optimization Approach With Statistical Guarantees

Net-Zero Scheduling of Multi-Energy Building Energy Systems: A Learning-Based Robust Optimization Approach With Statistical Guarantees

A Benchmark Proposal for Non‐Generative Fair Adversarial Learning Strategies Using a Fairness‐Utility Trade‐off Metric

ABSTRACTAI systems for decision‐making have become increasingly popular in several areas. However, it is possible to identify biased decisions in many applications, which have become a concern for the computer science, artificial intelligence, and law communities. Therefore, researchers are proposing solutions to mitigate bias and discrimination among decision‐makers. Some explored strategies are based on GANs to generate fair data. Others are based on adversarial learning to achieve fairness by encoding fairness constraints through an adversarial model. Moreover, it is usual for each proposal to assess its model with a specific metric, making comparing current approaches a complex task. Therefore, this work proposes a systematical benchmark procedure to assess the fair machine learning models. The proposed procedure comprises a fairness‐utility trade‐off metric (), the utility and fairness metrics to compose this assessment, the used datasets and preparation, and the statistical test. A previous work presents some of these definitions. The present work enriches the procedure by increasing the applied datasets and statistical guarantees when comparing the models' results. We performed this benchmark evaluation for the non‐generative adversarial models, analyzing the literature models from the same metric perspective. This assessment could not indicate a single model which better performs for all datasets. However, we built an understanding of how each model performs on each dataset with statistical confidence.

End-to-End Statistical Model Checking for Parameterization and Stability Analysis of ODE Models

We propose a simulation-based technique for the parameterization and the stability analysis of parametric Ordinary Differential Equations. This technique is an adaptation of Statistical Model Checking, often used to verify the validity of biological models, to the setting of Ordinary Differential Equations systems. The aim of our technique is to estimate the probability of satisfying a given property under the variability of the parameter or initial condition of the ODE, with any metrics of choice. To do so, we discretize the values space and use statistical model checking to evaluate each individual value w.r.t. provided data. Contrary to other existing methods, we provide statistical guarantees regarding our results that take into account the unavoidable approximation errors introduced through the numerical integration of the ODE system performed while simulating. In order to show the potential of our technique, we present its application to two case studies taken from the literature, one relative to the growth of a jellyfish population, and the other concerning a well-known oscillator model.

Leveraging conformal prediction to annotate enzyme function space with limited false positives.

Machine learning (ML) is increasingly being used to guide biological discovery in biomedicine such as prioritizing promising small molecules in drug discovery. In those applications, ML models are used to predict the properties of biological systems, and researchers use these predictions to prioritize candidates as new biological hypotheses for downstream experimental validations. However, when applied to unseen situations, these models can be overconfident and produce a large number of false positives. One solution to address this issue is to quantify the model's prediction uncertainty and provide a set of hypotheses with a controlled false discovery rate (FDR) pre-specified by researchers. We propose CPEC, an ML framework for FDR-controlled biological discovery. We demonstrate its effectiveness using enzyme function annotation as a case study, simulating the discovery process of identifying the functions of less-characterized enzymes. CPEC integrates a deep learning model with a statistical tool known as conformal prediction, providing accurate and FDR-controlled function predictions for a given protein enzyme. Conformal prediction provides rigorous statistical guarantees to the predictive model and ensures that the expected FDR will not exceed a user-specified level with high probability. Evaluation experiments show that CPEC achieves reliable FDR control, better or comparable prediction performance at a lower FDR than existing methods, and accurate predictions for enzymes under-represented in the training data. We expect CPEC to be a useful tool for biological discovery applications where a high yield rate in validation experiments is desired but the experimental budget is limited.

Memory-based Distribution Shift Detection for Learning Enabled Cyber-Physical Systems with Statistical Guarantees

Incorporating learning based components in the current state-of-the-art cyber-physical systems (CPS) has been a challenge due to the brittleness of the underlying deep neural networks. On the bright side, if executed correctly with safety guarantees, this has the ability to revolutionize domains like autonomous systems, medicine, and other safety-critical domains. This is because it would allow system designers to use high-dimensional outputs from sensors like camera and LiDAR. The trepidation in deploying systems with vision and LiDAR components comes from incidents of catastrophic failures in the real world. Recent reports of self-driving cars running into difficult to handle scenarios is ingrained in the software components which handle such sensor inputs. The ability to handle such high-dimensional signals is due to the explosion of algorithms which use deep neural networks. Sadly, the reason behind the safety issues is also due to deep neural networks themselves. The pitfalls occur due to possible over-fitting and lack of awareness about the blind spots induced by the training distribution. Ideally, system designers would wish to cover as many scenarios during training as possible. However, achieving a meaningful coverage is impossible. This naturally leads to the following question: is it feasible to flag out-of-distribution (OOD) samples without causing too many false alarms? Such an OOD detector should be executable in a fashion that is computationally efficient. This is because OOD detectors often are executed as frequently as the sensors are sampled. Our aim in this article is to build an effective anomaly detector. To this end, we propose the idea of a memory bank to cache data samples which are representative enough to cover most of the in-distribution data. The similarity with respect to such samples can be a measure of familiarity of the test input. This is made possible by an appropriate choice of distance function tailored to the type of sensor we are interested in. Additionally, we adapt conformal anomaly detection framework to capture the distribution shifts with a guarantee of false alarm rate. We report the performance of our technique on two challenging scenarios: a self-driving car setting implemented inside the simulator CARLA with image inputs and autonomous racing car navigation setting with LiDAR inputs. From the experiments, it is clear that a deviation from the in-distribution setting can potentially lead to unsafe behavior. It should be noted that not all OOD inputs lead to precarious situations in practice, but staying in-distribution is akin to staying within a safety bubble and predictable behavior. An added benefit of our memory-based approach is that the OOD detector produces interpretable feedback for a human designer. This is of utmost importance since it recommends a potential fix for the situation as well. In other competing approaches, such feedback is difficult to obtain due to reliance on techniques which use variational autoencoders.

Optimistic Rates: A Unifying Theory for Interpolation Learning and Regularization in Linear Regression

We study a localized notion of uniform convergence known as an “optimistic rate” [ 34 , 39 ] for linear regression with Gaussian data. Our refined analysis avoids the hidden constant and logarithmic factor in existing results, which are known to be crucial in high-dimensional settings, especially for understanding interpolation learning. As a special case, our analysis recovers the guarantee from Koehler et al. [ 21 ], which tightly characterizes the population risk of low-norm interpolators under the benign overfitting conditions. Our optimistic rate bound, though, also analyzes predictors with arbitrary training error. This allows us to recover some classical statistical guarantees for ridge and LASSO regression under random designs and helps us obtain a precise understanding of the excess risk of near-interpolators in the over-parameterized regime.

Variable Importance in High-Dimensional Settings Requires Grouping

Explaining the decision process of machine learning algorithms is nowadays crucial for both model’s performance enhancement and human comprehension. This can be achieved by assessing the variable importance of single variables, even for high-capacity non-linear methods, e.g. Deep Neural Networks (DNNs). While only removal-based approaches, such as Permutation Importance (PI), can bring statistical validity, they return misleading results when variables are correlated. Conditional Permutation Importance (CPI) bypasses PI’s limitations in such cases. However, in high-dimensional settings, where high correlations between the variables cancel their conditional importance, the use of CPI as well as other methods leads to unreliable results, besides prohibitive computation costs. Grouping variables statistically via clustering or some prior knowledge gains some power back and leads to better interpretations. In this work, we introduce BCPI (Block-Based Conditional Permutation Importance), a new generic framework for variable importance computation with statistical guarantees handling both single and group cases. Furthermore, as handling groups with high cardinality (such as a set of observations of a given modality) are both time-consuming and resource-intensive, we also introduce a new stacking approach extending the DNN architecture with sub-linear layers adapted to the group structure. We show that the ensuing approach extended with stacking controls the type-I error even with highly-correlated groups and shows top accuracy across benchmarks. Furthermore, we perform a real-world data analysis in a large-scale medical dataset where we aim to show the consistency between our results and the literature for a biomarker prediction.

MeshPointNet: 3D Surface Classification Using Graph Neural Networks and Conformal Predictions on Mesh-Based Representations

Abstract In many design automation applications, accurate segmentation and classification of 3D surfaces and extraction of geometric insight from 3D models can be pivotal. This paper primarily introduces a machine learning-based scheme that leverages graph neural networks for handling 3D geometries, specifically for surface classification. Our model demonstrates superior performance against two state-of-the-art models, PointNet + + and PointMLP, in terms of surface classification accuracy, beating both models. Central to our contribution is the novel incorporation of conformal predictions, a method that offers robust uncertainty quantification and handling with marginal statistical guarantees. Unlike traditional approaches, conformal predictions enable our model to ensure precision, especially in challenging scenarios where mistakes can be highly costly. This robustness proves invaluable in design applications, and as a case in point, we showcase its utility in automating the computational fluid dynamics meshing process for aircraft models based on expert guidance. Our results reveal that our automatically generated mesh, guided by the proposed rules by experts enabled through the segmentation model, is not only efficient but matches the quality of expert-generated meshes, leading to accurate simulations.

Statistical guaranteed noisy tensor recovery by fusing low-rankness on all orientations in frequency–original domains

Low-rank tensor recovery faces challenges in accurately defining the low-rankness of a tensor. Most existing definitions typically focus on one domain alone — either the original or frequency domain. Additionally, certain definitions often exhibit limitations in their sensitivity to orientation variation. To overcome these challenges, we define a novel tensor rank, the Orientation Invariant Hybrid Rank (OIHR). This rank fuses rank information across all orientations in both frequency and original domains. Employing its convex approximation, the Orientation Invariant Hybrid Nuclear Norm (OIHNN), we propose a general tensor recovery model. We further explore the statistical performance of the estimator based on this model, establishing a deterministic upper bound on the estimation error under generic noise. Furthermore, non-asymptotic upper bounds under Gaussian noise are separately derived for two specific cases: tensor compressive sensing and tensor completion. Finally, we propose the algorithm to solve the model. Extensive experiments on both synthetic and real data are conducted to validate the statistical guarantees and verify the effectiveness of our algorithm.

Identification and Auto-Debiased Machine Learning for Outcome-Conditioned Average Structural Derivatives

This study proposes a new class of heterogeneous causal quantities, referred to as outcome-conditioned average structural derivatives (OASDs), in a general nonseparable model. An OASD is the average partial effect of a marginal change in a continuous treatment on individuals located on different parts of an outcome distribution, irrespective of individuals’ characteristics. We show that OASDs extend the unconditional quantile partial effects (UQPE) proposed by Firpo, Fortin, and Lemieux to that conditional on a set of outcome values by effectively integrating the UQPE. Exploiting such relationship brings about two merits. First, unlike UQPE that is generally not n -estimable, OASD is shown to be n -estimable. Second, our estimator achieves semiparametric efficiency bound which is a new result in the literature. We propose a novel, automatic, debiased machine-learning estimator for an OASD, and present asymptotic statistical guarantees for it. The estimator is proven to be n -consistent, asymptotically normal, and semi-parametrically efficient. We also prove the validity of the bootstrap procedure for uniform inference for the OASD process. We apply the method to Imbens, Rubin, and Sacerdote’s lottery data.

Finding Near-optimal Configurations in Colossal Spaces with Statistical Guarantees

A Software Product Line ( SPL ) is a family of similar programs. Each program is defined by a unique set of features, called a configuration , that satisfies all feature constraints. “What configuration achieves the best performance for a given workload?” is the SPL Optimization ( SPLO ) challenge. SPLO is daunting: just 80 unconstrained features yield 10 24 unique configurations, which equals the estimated number of stars in the universe. We explain (a) how uniform random sampling and random search algorithms solve SPLO more efficiently and accurately than current machine-learned performance models and (b) how to compute statistical guarantees on the quality of a returned configuration; i.e., it is within x% of optimal with y% confidence.

Toward quantitative super-resolution microscopy: molecular maps with statistical guarantees.

Quantifying the number of molecules from fluorescence microscopy measurements is an important topic in cell biology and medical research. In this work, we present a consecutive algorithm for super-resolution (stimulated emission depletion (STED)) scanning microscopy that provides molecule counts in automatically generated image segments and offers statistical guarantees in form of asymptotic confidence intervals. To this end, we first apply a multiscale scanning procedure on STED microscopy measurements of the sample to obtain a system of significant regions, each of which contains at least one molecule with prescribed uniform probability. This system of regions will typically be highly redundant and consists of rectangular building blocks. To choose an informative but non-redundant subset of more naturally shaped regions, we hybridize our system with the result of a generic segmentation algorithm. The diameter of the segments can be of the order of the resolution of the microscope. Using multiple photon coincidence measurements of the same sample in confocal mode, we are then able to estimate the brightness and number of molecules and give uniform confidence intervals on the molecule counts for each previously constructed segment. In other words, we establish a so-called molecular map with uniform error control. The performance of the algorithm is investigated on simulated and real data.

Denoising cosine similarity: A theory-driven approach for efficient representation learning

Representation learning has been increasing its impact on the research and practice of machine learning, since it enables to learn representations that can apply to various downstream tasks efficiently. However, recent works pay little attention to the fact that real-world datasets used during the stage of representation learning are commonly contaminated by noise, which can degrade the quality of learned representations. This paper tackles the problem to learn robust representations against noise in a raw dataset. To this end, inspired by recent works on denoising and the success of the cosine-similarity-based objective functions in representation learning, we propose the denoising Cosine-Similarity (dCS) loss. The dCS loss is a modified cosine-similarity loss and incorporates a denoising property, which is supported by both our theoretical and empirical findings. To make the dCS loss implementable, we also construct the estimators of the dCS loss with statistical guarantees. Finally, we empirically show the efficiency of the dCS loss over the baseline objective functions in vision and speech domains.

Deep nonparametric estimation of intrinsic data structures by chart autoencoders: Generalization error and robustness

Autoencoders have demonstrated remarkable success in learning low-dimensional latent features of high-dimensional data across various applications. Assuming that data are sampled near a low-dimensional manifold, we employ chart autoencoders, which encode data into low-dimensional latent features on a collection of charts, preserving the topology and geometry of the data manifold. Our paper establishes statistical guarantees on the generalization error of chart autoencoders, and we demonstrate their denoising capabilities by considering n noisy training samples, along with their noise-free counterparts, on a d-dimensional manifold. By training autoencoders, we show that chart autoencoders can effectively denoise the input data with normal noise. We prove that, under proper network architectures, chart autoencoders achieve a squared generalization error in the order of n−2d+2log4⁡n, which depends on the intrinsic dimension of the manifold and only weakly depends on the ambient dimension and noise level. We further extend our theory on data with noise containing both normal and tangential components, where chart autoencoders still exhibit a denoising effect for the normal component. As a special case, our theory also applies to classical autoencoders, as long as the data manifold has a global parametrization. Our results provide a solid theoretical foundation for the effectiveness of autoencoders, which is further validated through several numerical experiments.

A Doubly Stochastic Simulator with Applications in Arrivals Modeling and Simulation

For several areas in the domain of operations research and management science, such as service, logistics and supply chain, and financial systems, the randomness of arrivals is one primary source of uncertainty. Appropriately modeling, statistically characterizing, and efficiently simulating the arrival processes are critical for policy and performance evaluation in the related systems. Classic Monte Carlo simulators have advantages in capturing the interpretable “physics” of a stochastic object, whereas neural network–based simulators have advantages in capturing less-interpretable complicated dependence within a high-dimensional distribution. In “A Doubly Stochastic Simulator with Applications in Arrivals Modeling and Simulation,” Zheng, Zheng, and Zhu propose a doubly stochastic simulator that integrates a stochastic generative neural network and a classic Monte Carlo Poisson simulator to utilize the advantages of both. They provide statistical guarantees and demonstrate empirical performances of the proposed methods.

PatchCensor: Patch Robustness Certification for Transformers via Exhaustive Testing

In the past few years, Transformer has been widely adopted in many domains and applications because of its impressive performance. Vision Transformer (ViT), a successful and well-known variant, attracts considerable attention from both industry and academia thanks to its record-breaking performance in various vision tasks. However, ViT is also highly nonlinear like other classical neural networks and could be easily fooled by both natural and adversarial perturbations. This limitation could pose a threat to the deployment of ViT in the real industrial environment, especially in safety-critical scenarios. How to improve the robustness of ViT is thus an urgent issue that needs to be addressed. Among all kinds of robustness, patch robustness is defined as giving a reliable output when a random patch in the input domain is perturbed. The perturbation could be natural corruption, such as part of the camera lens being blurred. It could also be a distribution shift, such as an object that does not exist in the training data suddenly appearing in the camera. And in the worst case, there could be a malicious adversarial patch attack that aims to fool the prediction of a machine learning model by arbitrarily modifying pixels within a restricted region of an input image. This kind of attack is also called physical attack, as it is believed to be more real than digital attack. Although there has been some work on patch robustness improvement of Convolutional Neural Network, related studies on its counterpart ViT are still at an early stage as ViT is usually much more complex with far more parameters. It is harder to assess and improve its robustness, not to mention to provide a provable guarantee. In this work, we propose PatchCensor, aiming to certify the patch robustness of ViT by applying exhaustive testing. We try to provide a provable guarantee by considering the worst patch attack scenarios. Unlike empirical defenses against adversarial patches that may be adaptively breached, certified robust approaches can provide a certified accuracy against arbitrary attacks under certain conditions. However, existing robustness certifications are mostly based on robust training, which often requires substantial training efforts and the sacrifice of model performance on normal samples. To bridge the gap, PatchCensor seeks to improve the robustness of the whole system by detecting abnormal inputs instead of training a robust model and asking it to give reliable results for every input, which may inevitably compromise accuracy. Specifically, each input is tested by voting over multiple inferences with different mutated attention masks, where at least one inference is guaranteed to exclude the abnormal patch. This can be seen as complete-coverage testing, which could provide a statistical guarantee on inference at the test time. Our comprehensive evaluation demonstrates that PatchCensor is able to achieve high certified accuracy (e.g., 67.1% on ImageNet for 2%-pixel adversarial patches), significantly outperforming state-of-the-art techniques while achieving similar clean accuracy (81.8% on ImageNet). The clean accuracy is the same as vanilla ViT models. Meanwhile, our technique also supports flexible configurations to handle different adversarial patch sizes by simply changing the masking strategy.