High-dimensional Sets Research Articles

HIGH-DIMENSIONAL FORECASTING WITH KNOWN KNOWNS AND KNOWN UNKNOWNS

Abstract Forecasts play a central role in decision-making under uncertainty. After a brief review of the general issues, this article considers ways of using high-dimensional data in forecasting. We consider selecting variables from a known active set, known knowns, using Lasso and One Covariate at a time Multiple Testing, and approximating unobserved latent factors, known unknowns, by various means. This combines both sparse and dense approaches to forecasting. We demonstrate the various issues involved in variable selection in a high-dimensional setting with an application to forecasting UK inflation at different horizons over the period 2020q1–2023q1. This application shows both the power of parsimonious models and the importance of allowing for global variables.

Causal effect estimation in survival analysis with high dimensional confounders.

With the ever advancing of modern technologies, it has become increasingly common that the number of collected confounders exceeds the number of subjects in a data set. However, matching based methods for estimating causal treatment effect in their original forms are not capable of handling high-dimensional confounders, and their various modified versions lack statistical support and valid inference tools. In this article, we propose a new approach for estimating causal treatment effect, defined as the difference of the restricted mean survival time (RMST) under different treatments in high-dimensional setting for survival data. We combine the factor model and the sufficient dimension reduction techniques to construct propensity score and prognostic score. Based on these scores, we develop a kernel based doubly robust estimator of the RMST difference. We demonstrate its link to matching and establish the consistency and asymptotic normality of the estimator. We illustrate our method by analyzing a dataset from a study aimed at comparing the effects of two alternative treatments on the RMST of patients with diffuse large B cell lymphoma.

Semi-Supervised Triply Robust Inductive Transfer Learning

In this work, we propose a Semi-supervised Triply Robust Inductive transFer LEarning (STRIFLE) approach, which integrates heterogeneous data from a label-rich source population and a label-scarce target population and uses a large amount of unlabeled data simultaneously to improve the learning accuracy in the target population. Specifically, we consider a high dimensional covariate shift setting and employ two nuisance models, a density ratio model and an imputation model, to combine transfer learning and surrogate-assisted semi-supervised learning strategies effectively and achieve triple robustness. While the STRIFLE approach assumes the target and source populations to share the same conditional distribution of outcome Y given both the surrogate features S and predictors X , it allows the true underlying model of Y⏧ X to differ between the two populations due to the potential covariate shift in S and X . Different from double robustness, even if both nuisance models are misspecified or the distribution of Y⏧ S , X is not the same between the two populations when the shifted source population and the target population share enough similarities, the triply robust STRIFLE estimator can still partially use the source population when the shifted source population and the target population share enough similarities. Moreover, it is guaranteed to be no worse than the target-only surrogate-assisted semi-supervised estimator with an additional error term from transferability detection. These desirable properties of our estimator are established theoretically and verified in finite samples via extensive simulation studies. We use the STRIFLE estimator to train a Type II diabetes polygenic risk prediction model for the African American target population by transferring knowledge from electronic health records linked genomic data observed in a larger European source population. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

A multi-objective robust dispatch strategy for renewable energy microgrids considering multiple uncertainties

The demand for low-carbon transformations and the uncertainty of renewable energy sources and loads present significant challenges for the optimal dispatch of microgrid. This study proposed a multi-objective robust dispatch strategy to reduce the risks associated with the uncertainty of renewable energy source output and loads while promoting low-carbon and economical microgrid operation. The economic emission dispatch problem for a microgrid was formulated as a multi-objective robust dual-layer optimization model. Consequently, a high-dimensional adjustable linear polyhedral uncertainty set was proposed to describe the uncertainty of renewable energy sources and loads. This study transformed the original model into an easy-to-solve single-layer second-order cone programming optimal power flow optimization model by employing second-order cone relaxation and duality transformation. Thereafter, a synthetic membership function was proposed to determine the optimal compromise solution. To determine the charging and discharging statuses of the battery storage system and the electricity traded between the microgrid and the external power grid, a battery storage system control strategy based on time-of-use electricity prices and real-time power flow calculations was proposed. Simulations conducted on a modified IEEE-30 bus system demonstrated that the proposed strategy effectively reduced the economic costs and carbon emissions of the microgrid by 8.23 % and 2.43 %, respectively.

Predicting Pseudomonas aeruginosa drug resistance using artificial intelligence and clinical MALDI-TOF mass spectra.

Matrix-assisted laser desorption/ionization-time of flight mass spectrometry (MALDI-TOF MS) is widely used in clinical microbiology laboratories for bacterial identification but its use for detection of antimicrobial resistance (AMR) remains limited. Here, we used MALDI-TOF MS with artificial intelligence (AI) approaches to successfully predict AMR in Pseudomonas aeruginosa, a priority pathogen with complex AMR mechanisms. The highest performance was achieved for modern β-lactam/β-lactamase inhibitor drugs, namely, ceftazidime/avibactam and ceftolozane/tazobactam. For these drugs, the model demonstrated area under the receiver operating characteristic curve (AUROC) of 0.869 and 0.856, specificity of 0.925 and 0.897, and sensitivity of 0.731 and 0.714, respectively. As part of this work, we developed dynamic binning, a feature engineering technique that effectively reduces the high-dimensional feature set and has wide-ranging applicability to MALDI-TOF MS data. Compared to conventional feature engineering approaches, the dynamic binning method yielded highest performance in 7 of 10 antimicrobials. Moreover, we showcased the efficacy of transfer learning in enhancing the AUROC performance for 8 of 11 antimicrobials. By assessing the contribution of features to the model's prediction, we identified proteins that may contribute to AMR mechanisms. Our findings demonstrate the potential of combining AI with MALDI-TOF MS as a rapid AMR diagnostic tool for Pseudomonas aeruginosa.IMPORTANCEPseudomonas aeruginosa is a key bacterial pathogen that causes significant global morbidity and mortality. Antimicrobial resistance (AMR) emerges rapidly in P. aeruginosa and is driven by complex mechanisms. Drug-resistant P. aeruginosa is a major challenge in clinical settings due to limited treatment options. Early detection of AMR can guide antibiotic choices, improve patient outcomes, and avoid unnecessary antibiotic use. Matrix-assisted laser desorption/ionization-time of flight mass spectrometry (MALDI-TOF MS) is widely used for rapid species identification in clinical microbiology. In this study, we repurposed mass spectra generated by MALDI-TOF and used them as inputs for artificial intelligence approaches to successfully predict AMR in P. aeruginosa for multiple key antibiotic classes. This work represents an important advance toward using MALDI-TOF as a rapid AMR diagnostic for P. aeruginosa in clinical settings.

Online graph topology learning from matrix-valued time series

The focus is on the statistical analysis of matrix-valued time series, where data is collected over a network of sensors, typically at spatial locations, over time. Each sensor records a vector of features at each time point, creating a vectorial time series for each sensor. The goal is to identify the dependency structure among these sensors and represent it with a graph. When only one feature per sensor is observed, vector auto-regressive (VAR) models are commonly used to infer Granger causality, resulting in a causal graph. The first contribution extends VAR models to matrix-variate models for the purpose of graph learning. Additionally, two online procedures are proposed for both low and high dimensions, enabling rapid updates of coefficient estimates as new samples arrive. In the high-dimensional setting, a novel Lasso-type approach is introduced, and homotopy algorithms are developed for online learning. An adaptive tuning procedure for the regularization parameter is also provided. Given that the application of auto-regressive models to data typically requires detrending, which is not feasible in an online context, the proposed AR models are augmented by incorporating trend as an additional parameter, with a particular focus on periodic trends. The online algorithms are adapted to these augmented data models, allowing for simultaneous learning of the graph and trend from streaming samples. Numerical experiments using both synthetic and real data demonstrate the effectiveness of the proposed methods.

Robust estimation for number of factors in high dimensional factor modeling via Spearman correlation matrix

Determining the number of factors in high-dimensional factor modeling is essential but challenging, especially when the data are heavy-tailed. In this paper, we introduce a new estimator based on the spectral properties of Spearman sample correlation matrix under the high-dimensional setting, where both dimension and sample size tend to infinity proportionally. Our estimator is robust against heavy tails in either the common factors or idiosyncratic errors. The consistency of our estimator is established under mild conditions. Numerical experiments demonstrate the superiority of our estimator compared to existing methods.

Knowledge-slanted random forest method for high-dimensional data and small sample size with a feature selection application for gene expression data

The use of prior knowledge in the machine learning framework has been considered a potential tool to handle the curse of dimensionality in genetic and genomics data. Although random forest (RF) represents a flexible non-parametric approach with several advantages, it can provide poor accuracy in high-dimensional settings, mainly in scenarios with small sample sizes. We propose a knowledge-slanted RF that integrates biological networks as prior knowledge into the model to improve its performance and explainability, exemplifying its use for selecting and identifying relevant genes. knowledge-slanted RF is a combination of two stages. First, prior knowledge represented by graphs is translated by running a random walk with restart algorithm to determine the relevance of each gene based on its connection and localization on a protein-protein interaction network. Then, each relevance is used to modify the selection probability to draw a gene as a candidate split-feature in the conventional RF. Experiments in simulated datasets with very small sample sizes (n≤30)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$(n \\le 30)$$\\end{document} comparing knowledge-slanted RF against conventional RF and logistic lasso regression, suggest an improved precision in outcome prediction compared to the other methods. The knowledge-slanted RF was completed with the introduction of a modified version of the Boruta feature selection algorithm. Finally, knowledge-slanted RF identified more relevant biological genes, offering a higher level of explainability for users than conventional RF. These findings were corroborated in one real case to identify relevant genes to calcific aortic valve stenosis.

The future of cosmological likelihood-based inference: accelerated high-dimensional parameter estimation and model comparison

We advocate for a new paradigm of cosmological likelihood-based inference, leveraging recent developments in machine learning and its underlying technology, to accelerate Bayesian inference in high-dimensional settings. Specifically, we combine (i) emulation, where a machine learning model is trained to mimic cosmological observables, e.g. CosmoPower-JAX; (ii) differentiable and probabilistic programming, e.g. JAX and NumPyro, respectively; (iii) scalable Markov chain Monte Carlo (MCMC) sampling techniques that exploit gradients, e.g. Hamiltonian Monte Carlo; and (iv) decoupled and scalable Bayesian model selection techniques that compute the Bayesian evidence purely from posterior samples, e.g. the learned harmonic mean implemented in harmonic. This paradigm allows us to carry out a complete Bayesian analysis, including both parameter estimation and model selection, in a fraction of the time of traditional approaches. First, we demonstrate the application of this paradigm on a simulated cosmic shear analysis for a Stage IV survey in 37- and 39-dimensional parameter spaces, comparing ΛCDM and a dynamical dark energy model ( w0waCDM). We recover posterior contours and evidence estimates that are in excellent agreement with those computed by the traditional nested sampling approach while reducing the computational cost from 8 months on 48 CPU cores to 2 days on 12 GPUs. Second, we consider a joint analysis between three simulated next-generation surveys, each performing a 3x2pt analysis, resulting in 157- and 159-dimensional parameter spaces. Standard nested sampling techniques are simply unlikely to be feasible in this high-dimensional setting, requiring a projected 12 years of compute time on 48 CPU cores; on the other hand, the proposed approach only requires 8 days of compute time on 24 GPUs. All packages used in our analyses are publicly available.

Random projection ensemble conformal prediction for high-dimensional classification

In classification problems, many models with superior performance fail to provide confidence estimates or intervals for each prediction. This lack of reliability poses risks in real-world applications, making these models difficult to trust. Conformal prediction, as distribution-free and model-free approaches with finite-sample coverage guarantee, have recently been widely used to construct prediction sets for classification models. However, traditional conformal prediction methods only produce set-valued results without specifying a definitive predicted class. Particularly in complex settings, these methods fail to assist models in effectively addressing challenges such as high dimensionality, resulting in ambiguous prediction sets with low statistical efficiency, i.e. the prediction sets contain many false classes. In this study, a novel Ensemble Conformal Prediction algorithm based on Random Projection and a designed voting strategy, RPECP, is developed to tackle these challenges. Initially, a procedure for selecting the approximately oracle random projections and classifiers is executed to best leverage the internal information and structure of the data. Subsequently, based on the approximately oracle random projections and underlying classifiers, conformal prediction is performed on new test samples in a lower-dimensional space, resulting in multiple independent prediction sets. Finally, an accurate predicted class and a precise prediction set with high coverage and statistical efficiency are produced through a designed voting strategy. Compared to several base classifiers, RPECP obtain higher classification accuracy; against other conformal prediction algorithms, it achieves less ambiguous prediction sets with fewer false classes while guaranteeing high coverage. For illustration, this paper demonstrates RPECP's superiority over other methods in four cases: two high-dimensional settings and two real-world datasets.

Selection of Tuning Parameter and Comparison of Lasso and Adaptive Lasso on ZYZ Condition: A Monte Carlo Study

One of the fundamental objectives of statistics is to achieve accurate predictions. In high-dimensional settings (where the number of variables, p, exceeds the number of observations, n), the performance of ordinary least squares (OLS) is often suboptimal due to its high variance, which leads to lower prediction accuracy. Shrinking the variables is a promising approach, and methods such as ridge regression, elastic net, lasso, and adaptive lasso are well-known techniques for this purpose. While variable shrinkage introduces a small bias, it significantly reduces the variance compared to OLS. The effectiveness of shrinkage methods largely depends on the selection of the tuning parameter. Cross-validation and the Bayesian Information Criterion (BIC) are commonly used for this purpose, and an improved version of BIC has shown impressive results.???????? = (5.6,5.6,5.6,0), ???????? = (3,1.5,0,0,2,0,0,0), ???????? = (0.85,0.85,0.85,0) are the multiple regression models which are compared.

A machine learning analysis of the value-added intellectual coefficient’s effect on firm performance

Purpose Recently, machine learning (ML) methods gained popularity in finance and accounting research as alternatives to econometric analysis. Their success in high-dimensional settings is promising as a cure for the shortcomings of econometric analysis. The purpose of this study is to prove further the relationship between intellectual capital (IC) efficiency and firm performance using ML methods. Design/methodology/approach This study used the double selection, partialing-out and cross-fit partialing-out LASSO estimators to analyze the IC efficiency’s linear and nonlinear effects on firm performance using a sample of 2,581 North American firms from 1999 to 2021. The value-added intellectual capital (VAIC) and its components are used as indicators of IC efficiency. Firm performance is measured by return on equity, return on assets and market-to-book ratio. Findings The findings revealed significant connections between IC measures and firm performance. First, the VAIC, as an aggregate measure, significantly impacts both firm profitability and value. When the VAIC is decomposed into its breakdowns, it is revealed that structural capital efficiency substantially affects firm value, and capital employed efficiency has the same function for firm profitability. In contrast to the prevalent belief in the area, human capital efficiency’s impact is found to be less important than the others. Nonlinearities are also detected in the relationships. Originality/value As ML tools are most recently introduced to the IC literature, only a few studies have used them to expand the current knowledge. However, none of these studies investigated the role of IC as a determinant of firm performance. The present study fills this gap in the literature by investigating the effect of IC efficiency on firm performance using supervised ML methods. It also provides a novel approach by comparing the estimation results of three LASSO estimators. To the best of the author’s knowledge, this is the first study that has used LASSO in IC research.

Minimax rates of convergence for sliced inverse regression with differential privacy

Sliced inverse regression (SIR) is a highly efficient paradigm used for the purpose of dimension reduction by replacing high-dimensional covariates with a limited number of linear combinations. This paper focuses on the implementation of the classical SIR approach integrated with a Gaussian differential privacy mechanism to estimate the central space while preserving privacy. We illustrate the tradeoff between statistical accuracy and privacy in sufficient dimension reduction problems under both the classical low- dimensional and modern high-dimensional settings. Additionally, we achieve the minimax rate of the proposed estimator with Gaussian differential privacy constraint and illustrate that this rate is also optimal for multiple index models with bounded dimension of the central space. Extensive numerical studies on synthetic data sets are conducted to assess the effectiveness of the proposed technique in finite sample scenarios, and a real data analysis is presented to showcase its practical application.

Interplay between depth and width for interpolation in neural ODEs

Neural ordinary differential equations have emerged as a natural tool for supervised learning from a control perspective, yet a complete understanding of the role played by their architecture remains elusive. In this work, we examine the interplay between the width p and the number of transitions between layers L (corresponding to a depth of L+1). Specifically, we construct explicit controls interpolating either a finite dataset D, comprising N pairs of points in Rd, or two probability measures within a Wasserstein error margin ɛ>0. Our findings reveal a balancing trade-off between p and L, with L scaling as 1+O(N/p) for data interpolation, and as 1+Op−1+(1+p)−1ɛ−d for measures.In the high-dimensional and wide setting where d,p>N, our result can be refined to achieve L=0. This naturally raises the problem of data interpolation in the autonomous regime, characterized by L=0. We adopt two alternative approaches: either controlling in a probabilistic sense, or by relaxing the target condition. In the first case, when p=N we develop an inductive control strategy based on a separability assumption whose probability increases with d. In the second one, we establish an explicit error decay rate with respect to p which results from applying a universal approximation theorem to a custom-built Lipschitz vector field interpolating D.

Health impacts of wildfire smoke in California: Creating vulnerability profiles using machine learning methods in high-dimensional settings

Health impacts of wildfire smoke in California: Creating vulnerability profiles using machine learning methods in high-dimensional settings

Scalable Bayesian uncertainty quantification with data-driven priors for radio interferometric imaging

Abstract Next-generation radio interferometers like the Square Kilometer Array have the potential to unlock scientific discoveries thanks to their unprecedented angular resolution and sensitivity. One key to unlocking their potential resides in handling the deluge and complexity of incoming data. This challenge requires building radio interferometric (RI) imaging methods that can cope with the massive data sizes and provide high-quality image reconstructions with uncertainty quantification (UQ). This work proposes a method coined QuantifAI to address UQ in RI imaging with data-driven (learned) priors for high-dimensional settings. Our model, rooted in the Bayesian framework, uses a physically motivated model for the likelihood. The model exploits a data-driven convex prior potential, which can encode complex information learned implicitly from simulations and guarantee the log-concavity of the posterior. We leverage probability concentration phenomena of high-dimensional log-concave posteriors to obtain information about the posterior, avoiding MCMC sampling techniques. We rely on convex optimisation methods to compute the MAP estimation, which is known to be faster and better scale with dimension than MCMC strategies. QuantifAI allows us to compute local credible intervals and perform hypothesis testing of structure on the reconstructed image. We propose a novel fast method to compute pixel-wise uncertainties at different scales, which uses 3 and 6 orders of magnitude less likelihood evaluations than other UQ methods like length of the credible intervals and Monte Carlo posterior sampling, respectively. We demonstrate our method by reconstructing RI images in a simulated setting and carrying out fast and scalable UQ, which we validate with MCMC sampling. Our method shows an improved image quality and more meaningful uncertainties than the benchmark method based on a sparsity-promoting prior. QuantifAI’s source code is available from https://github.com/astro-informatics/QuantifAI.

Identifying payable cluster distributions for improved reservoir characterization: a robust unsupervised ML strategy for rock typing of depositional facies in heterogeneous rocks

The oil and gas industry relies on accurately predicting profitable clusters in subsurface formations for geophysical reservoir analysis. It is challenging to predict payable clusters in complicated geological settings like the Lower Indus Basin, Pakistan. In complex, high-dimensional heterogeneous geological settings, traditional statistical methods seldom provide correct results. Therefore, this paper introduces a robust unsupervised AI strategy designed to identify and classify profitable zones using self-organizing maps (SOM) and K-means clustering techniques. Results of SOM and K-means clustering provided the reservoir potentials of six depositional facies types (MBSD, DCSD, MBSMD, SSiCL, SMDFM, MBSh) based on cluster distributions. The depositional facies MBSD and DCSD exhibited high similarity and achieved a maximum effective porosity (PHIE) value of ≥ 15%, indicating good reservoir rock typing (RRT) features. The density-based spatial clustering of applications with noise (DBSCAN) showed minimum outliers through meta cluster attributes and confirmed the reliability of the generated cluster results. Shapley Additive Explanations (SHAP) model identified PHIE as the most significant parameter and was beneficial in identifying payable and non-payable clustering zones. Additionally, this strategy highlights the importance of unsupervised AI in managing profitable cluster distribution across various geological formations, going beyond simple reservoir characterization.

Enhanced Diabetes Prediction and Monitoring through Integration of Multimodal Data Using XGBoost

Abstract: Diabetes is a critical and become more complicated disease that can cause serious health problems if it is not adequately managed. The early diagnosis and treatment of diabetes is a critical component of the condition that can be greatly aided by data analysis and predictive algorithms. Through the use of data mining techniques, such as classification and prediction models, it is possible to analyse various elements of diabetes data and extract useful information that can be used for the early detection and prediction of the condition. One machine learning technique that can effectively and highly precisely predict diabetes is the XGBoost classifier. This method makes use of the gradient-boosting architecture and can handle large and intricate datasets with independent high-dimensional feature sets. Conversely, it is crucial to remember that the choice of the best algorithm for diabetes prediction could depend on the specifics of the data as well as the area of study being investigated. Data analysis and prediction methods can be applied not only to anticipate diabetes but also to monitor the disease's progression, find risk factors for diabetes and its complications, and assess the effectiveness of treatment. By using these techniques, medical professionals can obtain important insights into the disease's underlying causes, which helps them make informed decisions about patient management. The early detection and management of diabetes, a chronic disease that is rapidly expanding and poses major health risks, has the potential to be significantly improved through the application of data analysis and prediction algorithms. An accuracy rate of 89% was achieved by the XGBoost classifier, which demonstrated the highest level of performance.

Optimized Tail Bounds for Random Matrix Series.

Random matrix series are a significant component of random matrix theory, offering rich theoretical content and broad application prospects. In this paper, we propose modified versions of tail bounds for random matrix series, including matrix Gaussian (or Rademacher) and sub-Gaussian and infinitely divisible (i.d.) series. Unlike present studies, our results depend on the intrinsic dimension instead of ambient dimension. In some cases, the intrinsic dimension is much smaller than ambient dimension, which makes the modified versions suitable for high-dimensional or infinite-dimensional setting possible. In addition, we obtain the expectation bounds for random matrix series based on the intrinsic dimension.

Group lasso priors for Bayesian accelerated failure time models with left-truncated and interval-censored data.

An important task in health research is to characterize time-to-event outcomes such as disease onset or mortality in terms of a potentially high-dimensional set of risk factors. For example, prospective cohort studies of Alzheimer's disease (AD) typically enroll older adults for observation over several decades to assess the long-term impact of genetic and other factors on cognitive decline and mortality. The accelerated failure time model is particularly well-suited to such studies, structuring covariate effects as "horizontal" changes to the survival quantiles that conceptually reflect shifts in the outcome distribution due to lifelong exposures. However, this modeling task is complicated by the enrollment of adults at differing ages, and intermittent follow-up visits leading to interval-censored outcome information. Moreover, genetic and clinical risk factors are not only high-dimensional, but characterized by underlying grouping structures, such as by function or gene location. Such grouped high-dimensional covariates require shrinkage methods that directly acknowledge this structure to facilitate variable selection and estimation. In this paper, we address these considerations directly by proposing a Bayesian accelerated failure time model with a group-structured lasso penalty, designed for left-truncated and interval-censored time-to-event data. We develop an R package with a Markov chain Monte Carlo sampler for estimation. We present a simulation study examining the performance of this method relative to an ordinary lasso penalty and apply the proposed method to identify groups of predictive genetic and clinical risk factors for AD in the Religious Orders Study and Memory and Aging Project prospective cohort studies of AD and dementia.