High-dimensional Sets Research Articles

High-dimensional experiments for the downward continuation using the LRFMP algorithm

AbstractTime-dependent gravity data from satellite missions like GRACE-FO reveal mass redistribution in the system Earth at various time scales: long-term climate change signals, inter-annual phenomena like El Niño, seasonal mass transports and transients, e. g. due to earthquakes. For this contemporary issue, a classical inverse problem has to be considered: the gravitational potential has to be modelled on the Earth’s surface from measurements in space. This is also known as the downward continuation problem. Thus, it is important to further develop current mathematical methods for such inverse problems. For this, the (Learning) Inverse Problem Matching Pursuits ((L)IPMPs) have been developed within the last decade. Their unique feature is the combination of local as well as global trial functions in the approximative solution of an inverse problem such as the downward continuation of the gravitational potential. In this way, they harmonize the ideas of a traditional spherical harmonic ansatz and the radial basis function approach. Previous publications on these methods showed proofs of concept. In this paper, we report on the progress of our developments towards more realistic scenarios. In particular, we consider the methods for high-dimensional experiment settings with more than 500 000 grid points which yields a resolution of 20 km at best on a realistic satellite geometry. We also explain the changes in the methods that had to be done to work with such a large amount of data. The corresponding code (updated for big data use) is available at https://doi.org/10.5281/zenodo.8223771 under the licence CC BY-NC-SA 3.0 Germany.

Precision Least Squares: Estimation and Inference in High-Dimensions

The least squares estimator can be cast as depending only on the precision matrix of the data, similar to the weights of a global minimum variance portfolio. We give conditions under which any plug-in precision matrix estimator produces an unbiased and consistent least squares estimator for stationary time series regressions, in both low- and high-dimensional settings. Such conditions define a class of “Precision Least Squares” (PrLS) estimators, which are shown to be approximately Gaussian, efficient, and to provide automatic family-wise error control in large samples. For estimating high-dimensional sparse regression models, we propose a LASSO Cholesky estimator of the plug-in precision matrix. We show its consistency and how to properly bias correct it, thereby obtaining a LASSO Cholesky-based PrLS (LC-PrLS) estimator. LC-PrLS performs well in finite samples and better than state-of-the-art high-dimensional estimators. We employ LC-PrLS to investigate the dynamic network of predictive connections among a large set of global bank stock returns. We find that crisis years correspond to a collapse of predictive linkages.

Quantitative MRI Assessment of Post-Surgical Spinal Cord Injury Through Radiomic Analysis

Purpose: This study investigates radiomic efficacy in post-surgical traumatic spinal cord injury (SCI), overcoming MRI limitations from metal artifacts to enhance diagnosis, severity assessment, and lesion characterization or prognosis and therapy guidance. Background: Traumatic spinal cord injury (SCI) causes severe neurological deficits. While MRI allows qualitative injury evaluation, standard imaging alone has limitations for precise SCI diagnosis, severity stratification, and pathology characterization, which are needed to guide prognosis and therapy. Radiomics enables quantitative tissue phenotyping by extracting a high-dimensional set of descriptive texture features from medical images. However, the efficacy of postoperative radiomic quantification in the presence of metal-induced MRI artifacts from spinal instrumentation has yet to be fully explored. Methods: A total of 50 healthy controls and 12 SCI patients post-stabilization surgery underwent 3D multi-spectral MRI. Automated spinal cord segmentation was followed by radiomic feature extraction. Supervised machine learning categorized SCI versus controls, injury severity, and lesion location relative to instrumentation. Results: Radiomics differentiated SCI patients (Matthews correlation coefficient (MCC) 0.97; accuracy 1.0), categorized injury severity (MCC: 0.95; ACC: 0.98), and localized lesions (MCC: 0.85; ACC: 0.90). Combined T1 and T2 features outperformed individual modalities across tasks with gradient boosting models showing the highest efficacy. Conclusions: The radiomic framework achieved excellent performance, differentiating SCI from controls and accurately categorizing injury severity. The ability to reliably quantify SCI severity and localization could potentially inform diagnosis, prognosis, and guide therapy. Further research is warranted to validate radiomic SCI biomarkers and explore clinical integration.

Inference methods for multivariate coefficient of variation: a novel NPC-based approach

In many fields, from psychology and biology to quality control, logistics and risk management, the coefficient of variation (CV) and in particular the multivariate coefficient of variation (MCV), can be applied as measures of variability. Consequently, several parametric and nonparametric inference methods for CV and MCV have been introduced in the literature. These include a recently introduced permutation-based James-type studentized statistic for making inferences about MCV. In this article, we propose a new permutation-based solution exploiting the Non Parametric Combination (NPC) methodology. This new extension presents some major advantages: it can be applied to test both the one and two-sided ordered alternatives, it can deal with high-dimensional settings, even in the case of small sample sizes, and it can be used to deal with isotonic inference problems. In this paper, we present an extensive simulation study to address both one-way Multivariate Analysis of Variance (MANOVA)-like problems and isotonic inference problems. A case study is then provided to show a real data application of the new NPC-based solution.

Revisiting model complexity: Space-time correction of high dimensional variable sets in climate model simulations

Multivariate bias correction (BC) models are well-known to correct more statistical attributes in climate model simulations. However, their inherent complexity and excessive parameters can introduce higher uncertainty into future climate simulations. In contrast, univariate BC models, with fewer parameters, are limited to correcting certain attributes. An issue that has not been investigated in-depth is the impact ofan increased number of variables in the multivariate BC has on the bias-corrected climate models’ stability. This study compares the performance of a multivariate BC approach, Multivariate Recursive Nested Bias Correction (MRNBC), and a univariate BC approach, Continuous Wavelet-based Bias Correction (CWBC), as the number of variables to be corrected increases, known as the “curse of dimensionality” (CoD). The analysis uses high-resolution climate model outputs for both current and future simulations of sea surface temperature and precipitation in the Niño 3.4 region. Results show both BC models effectively correct current climate biases. As the number of variables increases, CWBC remains robust and produces sensible future simulations, while MRNBC’s complexity leads to deterioration in standard deviations and spatial cross-correlation. CWBC, based on univariate correction, isrelatively unaffected by the CoD.

A Neural Phillips Curve and a Deep Output Gap

Many problems plague empirical Phillips curves (PCs). Among them is the hurdle that the two key components, inflation expectations and the output gap, are both unobserved. Traditional remedies include proxying for the absentees or extracting them via assumptions-heavy filtering procedures. I propose an alternative route: a Hemisphere Neural Network (HNN) whose architecture yields a final layer where components can be interpreted as latent states within a Neural PC. First, HNN conducts the supervised estimation of nonlinearities that arise when translating a high-dimensional set of observed regressors into latent states. Second, forecasts are economically interpretable. Among other findings, the contribution of real activity to inflation appears understated in traditional PCs. In contrast, HNN captures the 2021 upswing in inflation and attributes it to a large positive output gap starting from late 2020. The unique path of HNN’s gap comes from dispensing with unemployment and GDP in favor of an amalgam of nonlinearly processed alternative tightness indicators.

Robust Estimation for Number of Factors in High Dimensional Factor Modeling via Spearman Correlation Matrix

Determining the number of factors in high-dimensional factor modeling is essential but challenging, especially when the data are heavy-tailed. In this article, we introduce a new estimator based on the spectral properties of Spearman sample correlation matrix under the high-dimensional setting, where both dimension and sample size tend to infinity proportionally. Our estimator is robust against heavy tails in either the common factors or idiosyncratic errors. The consistency of our estimator is established under mild conditions. Numerical experiments demonstrate the superiority of our estimator compared to existing methods. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

A Kernel approach for extending nonparametric multivariate analysis of variance in high-dimensional settings

The nonparametric multivariate analysis of variance (NPMANOVA) testing procedure has been proven to be a valuable tool for comparing groups. In the present paper, we propose a kernel extension of this technique in order to effectively confront high-dimensionality, a recurrent problem in many fields of science. The new method is called kernel multivariate analysis of variance (KMANOVA). The basic idea is to take advantage of the kernel framework: we propose to project the data from the original data space to a Hilbert space generated by a given kernel function and then perform the NPMANOVA method in the reproducing kernel Hilbert space (RKHS). Dispersion of the embedded points can be measured by the distance induced by the inner product in the RKHS but also by many other distances best suited in high-dimensional settings. For this purpose, we study two promising distances: a Manhattan-type distance and a distance based on an orthogonal projection of the embedded points in the direction of the group centroids. We show that the NPMANOVA method and the KMANOVA method with the induced distance are essentially equivalent. We also show that the KMANOVA method with the other two distances performs considerably better than the NPMANOVA method. We illustrate the advantages of our approach in the context of genetic association studies and demonstrate its usefulness on Alzheimer’s disease data. We also provide a software implementation of the method that is available on GitHub https://github.com/8699vicente/Kmanova.

A multi-objective robust dispatch strategy for renewable energy microgrids considering multiple uncertainties

The demand for low-carbon transformations and the uncertainty of renewable energy sources and loads present significant challenges for the optimal dispatch of microgrid. This study proposed a multi-objective robust dispatch strategy to reduce the risks associated with the uncertainty of renewable energy source output and loads while promoting low-carbon and economical microgrid operation. The economic emission dispatch problem for a microgrid was formulated as a multi-objective robust dual-layer optimization model. Consequently, a high-dimensional adjustable linear polyhedral uncertainty set was proposed to describe the uncertainty of renewable energy sources and loads. This study transformed the original model into an easy-to-solve single-layer second-order cone programming optimal power flow optimization model by employing second-order cone relaxation and duality transformation. Thereafter, a synthetic membership function was proposed to determine the optimal compromise solution. To determine the charging and discharging statuses of the battery storage system and the electricity traded between the microgrid and the external power grid, a battery storage system control strategy based on time-of-use electricity prices and real-time power flow calculations was proposed. Simulations conducted on a modified IEEE-30 bus system demonstrated that the proposed strategy effectively reduced the economic costs and carbon emissions of the microgrid by 8.23 % and 2.43 %, respectively.

HIGH-DIMENSIONAL FORECASTING WITH KNOWN KNOWNS AND KNOWN UNKNOWNS

Abstract Forecasts play a central role in decision-making under uncertainty. After a brief review of the general issues, this article considers ways of using high-dimensional data in forecasting. We consider selecting variables from a known active set, known knowns, using Lasso and One Covariate at a time Multiple Testing, and approximating unobserved latent factors, known unknowns, by various means. This combines both sparse and dense approaches to forecasting. We demonstrate the various issues involved in variable selection in a high-dimensional setting with an application to forecasting UK inflation at different horizons over the period 2020q1–2023q1. This application shows both the power of parsimonious models and the importance of allowing for global variables.

Causal effect estimation in survival analysis with high dimensional confounders.

With the ever advancing of modern technologies, it has become increasingly common that the number of collected confounders exceeds the number of subjects in a data set. However, matching based methods for estimating causal treatment effect in their original forms are not capable of handling high-dimensional confounders, and their various modified versions lack statistical support and valid inference tools. In this article, we propose a new approach for estimating causal treatment effect, defined as the difference of the restricted mean survival time (RMST) under different treatments in high-dimensional setting for survival data. We combine the factor model and the sufficient dimension reduction techniques to construct propensity score and prognostic score. Based on these scores, we develop a kernel based doubly robust estimator of the RMST difference. We demonstrate its link to matching and establish the consistency and asymptotic normality of the estimator. We illustrate our method by analyzing a dataset from a study aimed at comparing the effects of two alternative treatments on the RMST of patients with diffuse large B cell lymphoma.

Semi-Supervised Triply Robust Inductive Transfer Learning

In this work, we propose a Semi-supervised Triply Robust Inductive transFer LEarning (STRIFLE) approach, which integrates heterogeneous data from a label-rich source population and a label-scarce target population and uses a large amount of unlabeled data simultaneously to improve the learning accuracy in the target population. Specifically, we consider a high dimensional covariate shift setting and employ two nuisance models, a density ratio model and an imputation model, to combine transfer learning and surrogate-assisted semi-supervised learning strategies effectively and achieve triple robustness. While the STRIFLE approach assumes the target and source populations to share the same conditional distribution of outcome Y given both the surrogate features S and predictors X , it allows the true underlying model of Y⏧ X to differ between the two populations due to the potential covariate shift in S and X . Different from double robustness, even if both nuisance models are misspecified or the distribution of Y⏧ S , X is not the same between the two populations when the shifted source population and the target population share enough similarities, the triply robust STRIFLE estimator can still partially use the source population when the shifted source population and the target population share enough similarities. Moreover, it is guaranteed to be no worse than the target-only surrogate-assisted semi-supervised estimator with an additional error term from transferability detection. These desirable properties of our estimator are established theoretically and verified in finite samples via extensive simulation studies. We use the STRIFLE estimator to train a Type II diabetes polygenic risk prediction model for the African American target population by transferring knowledge from electronic health records linked genomic data observed in a larger European source population. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Predicting Pseudomonas aeruginosa drug resistance using artificial intelligence and clinical MALDI-TOF mass spectra.

Matrix-assisted laser desorption/ionization-time of flight mass spectrometry (MALDI-TOF MS) is widely used in clinical microbiology laboratories for bacterial identification but its use for detection of antimicrobial resistance (AMR) remains limited. Here, we used MALDI-TOF MS with artificial intelligence (AI) approaches to successfully predict AMR in Pseudomonas aeruginosa, a priority pathogen with complex AMR mechanisms. The highest performance was achieved for modern β-lactam/β-lactamase inhibitor drugs, namely, ceftazidime/avibactam and ceftolozane/tazobactam. For these drugs, the model demonstrated area under the receiver operating characteristic curve (AUROC) of 0.869 and 0.856, specificity of 0.925 and 0.897, and sensitivity of 0.731 and 0.714, respectively. As part of this work, we developed dynamic binning, a feature engineering technique that effectively reduces the high-dimensional feature set and has wide-ranging applicability to MALDI-TOF MS data. Compared to conventional feature engineering approaches, the dynamic binning method yielded highest performance in 7 of 10 antimicrobials. Moreover, we showcased the efficacy of transfer learning in enhancing the AUROC performance for 8 of 11 antimicrobials. By assessing the contribution of features to the model's prediction, we identified proteins that may contribute to AMR mechanisms. Our findings demonstrate the potential of combining AI with MALDI-TOF MS as a rapid AMR diagnostic tool for Pseudomonas aeruginosa.IMPORTANCEPseudomonas aeruginosa is a key bacterial pathogen that causes significant global morbidity and mortality. Antimicrobial resistance (AMR) emerges rapidly in P. aeruginosa and is driven by complex mechanisms. Drug-resistant P. aeruginosa is a major challenge in clinical settings due to limited treatment options. Early detection of AMR can guide antibiotic choices, improve patient outcomes, and avoid unnecessary antibiotic use. Matrix-assisted laser desorption/ionization-time of flight mass spectrometry (MALDI-TOF MS) is widely used for rapid species identification in clinical microbiology. In this study, we repurposed mass spectra generated by MALDI-TOF and used them as inputs for artificial intelligence approaches to successfully predict AMR in P. aeruginosa for multiple key antibiotic classes. This work represents an important advance toward using MALDI-TOF as a rapid AMR diagnostic for P. aeruginosa in clinical settings.

Online graph topology learning from matrix-valued time series

The focus is on the statistical analysis of matrix-valued time series, where data is collected over a network of sensors, typically at spatial locations, over time. Each sensor records a vector of features at each time point, creating a vectorial time series for each sensor. The goal is to identify the dependency structure among these sensors and represent it with a graph. When only one feature per sensor is observed, vector auto-regressive (VAR) models are commonly used to infer Granger causality, resulting in a causal graph. The first contribution extends VAR models to matrix-variate models for the purpose of graph learning. Additionally, two online procedures are proposed for both low and high dimensions, enabling rapid updates of coefficient estimates as new samples arrive. In the high-dimensional setting, a novel Lasso-type approach is introduced, and homotopy algorithms are developed for online learning. An adaptive tuning procedure for the regularization parameter is also provided. Given that the application of auto-regressive models to data typically requires detrending, which is not feasible in an online context, the proposed AR models are augmented by incorporating trend as an additional parameter, with a particular focus on periodic trends. The online algorithms are adapted to these augmented data models, allowing for simultaneous learning of the graph and trend from streaming samples. Numerical experiments using both synthetic and real data demonstrate the effectiveness of the proposed methods.

Knowledge-slanted random forest method for high-dimensional data and small sample size with a feature selection application for gene expression data

The use of prior knowledge in the machine learning framework has been considered a potential tool to handle the curse of dimensionality in genetic and genomics data. Although random forest (RF) represents a flexible non-parametric approach with several advantages, it can provide poor accuracy in high-dimensional settings, mainly in scenarios with small sample sizes. We propose a knowledge-slanted RF that integrates biological networks as prior knowledge into the model to improve its performance and explainability, exemplifying its use for selecting and identifying relevant genes. knowledge-slanted RF is a combination of two stages. First, prior knowledge represented by graphs is translated by running a random walk with restart algorithm to determine the relevance of each gene based on its connection and localization on a protein-protein interaction network. Then, each relevance is used to modify the selection probability to draw a gene as a candidate split-feature in the conventional RF. Experiments in simulated datasets with very small sample sizes (n≤30)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$(n \\le 30)$$\\end{document} comparing knowledge-slanted RF against conventional RF and logistic lasso regression, suggest an improved precision in outcome prediction compared to the other methods. The knowledge-slanted RF was completed with the introduction of a modified version of the Boruta feature selection algorithm. Finally, knowledge-slanted RF identified more relevant biological genes, offering a higher level of explainability for users than conventional RF. These findings were corroborated in one real case to identify relevant genes to calcific aortic valve stenosis.

The future of cosmological likelihood-based inference: accelerated high-dimensional parameter estimation and model comparison

We advocate for a new paradigm of cosmological likelihood-based inference, leveraging recent developments in machine learning and its underlying technology, to accelerate Bayesian inference in high-dimensional settings. Specifically, we combine (i) emulation, where a machine learning model is trained to mimic cosmological observables, e.g. CosmoPower-JAX; (ii) differentiable and probabilistic programming, e.g. JAX and NumPyro, respectively; (iii) scalable Markov chain Monte Carlo (MCMC) sampling techniques that exploit gradients, e.g. Hamiltonian Monte Carlo; and (iv) decoupled and scalable Bayesian model selection techniques that compute the Bayesian evidence purely from posterior samples, e.g. the learned harmonic mean implemented in harmonic. This paradigm allows us to carry out a complete Bayesian analysis, including both parameter estimation and model selection, in a fraction of the time of traditional approaches. First, we demonstrate the application of this paradigm on a simulated cosmic shear analysis for a Stage IV survey in 37- and 39-dimensional parameter spaces, comparing ΛCDM and a dynamical dark energy model ( w0waCDM). We recover posterior contours and evidence estimates that are in excellent agreement with those computed by the traditional nested sampling approach while reducing the computational cost from 8 months on 48 CPU cores to 2 days on 12 GPUs. Second, we consider a joint analysis between three simulated next-generation surveys, each performing a 3x2pt analysis, resulting in 157- and 159-dimensional parameter spaces. Standard nested sampling techniques are simply unlikely to be feasible in this high-dimensional setting, requiring a projected 12 years of compute time on 48 CPU cores; on the other hand, the proposed approach only requires 8 days of compute time on 24 GPUs. All packages used in our analyses are publicly available.

Random projection ensemble conformal prediction for high-dimensional classification

In classification problems, many models with superior performance fail to provide confidence estimates or intervals for each prediction. This lack of reliability poses risks in real-world applications, making these models difficult to trust. Conformal prediction, as distribution-free and model-free approaches with finite-sample coverage guarantee, have recently been widely used to construct prediction sets for classification models. However, traditional conformal prediction methods only produce set-valued results without specifying a definitive predicted class. Particularly in complex settings, these methods fail to assist models in effectively addressing challenges such as high dimensionality, resulting in ambiguous prediction sets with low statistical efficiency, i.e. the prediction sets contain many false classes. In this study, a novel Ensemble Conformal Prediction algorithm based on Random Projection and a designed voting strategy, RPECP, is developed to tackle these challenges. Initially, a procedure for selecting the approximately oracle random projections and classifiers is executed to best leverage the internal information and structure of the data. Subsequently, based on the approximately oracle random projections and underlying classifiers, conformal prediction is performed on new test samples in a lower-dimensional space, resulting in multiple independent prediction sets. Finally, an accurate predicted class and a precise prediction set with high coverage and statistical efficiency are produced through a designed voting strategy. Compared to several base classifiers, RPECP obtain higher classification accuracy; against other conformal prediction algorithms, it achieves less ambiguous prediction sets with fewer false classes while guaranteeing high coverage. For illustration, this paper demonstrates RPECP's superiority over other methods in four cases: two high-dimensional settings and two real-world datasets.

Selection of Tuning Parameter and Comparison of Lasso and Adaptive Lasso on ZYZ Condition: A Monte Carlo Study

One of the fundamental objectives of statistics is to achieve accurate predictions. In high-dimensional settings (where the number of variables, p, exceeds the number of observations, n), the performance of ordinary least squares (OLS) is often suboptimal due to its high variance, which leads to lower prediction accuracy. Shrinking the variables is a promising approach, and methods such as ridge regression, elastic net, lasso, and adaptive lasso are well-known techniques for this purpose. While variable shrinkage introduces a small bias, it significantly reduces the variance compared to OLS. The effectiveness of shrinkage methods largely depends on the selection of the tuning parameter. Cross-validation and the Bayesian Information Criterion (BIC) are commonly used for this purpose, and an improved version of BIC has shown impressive results.???????? = (5.6,5.6,5.6,0), ???????? = (3,1.5,0,0,2,0,0,0), ???????? = (0.85,0.85,0.85,0) are the multiple regression models which are compared.

A machine learning analysis of the value-added intellectual coefficient’s effect on firm performance

Purpose Recently, machine learning (ML) methods gained popularity in finance and accounting research as alternatives to econometric analysis. Their success in high-dimensional settings is promising as a cure for the shortcomings of econometric analysis. The purpose of this study is to prove further the relationship between intellectual capital (IC) efficiency and firm performance using ML methods. Design/methodology/approach This study used the double selection, partialing-out and cross-fit partialing-out LASSO estimators to analyze the IC efficiency’s linear and nonlinear effects on firm performance using a sample of 2,581 North American firms from 1999 to 2021. The value-added intellectual capital (VAIC) and its components are used as indicators of IC efficiency. Firm performance is measured by return on equity, return on assets and market-to-book ratio. Findings The findings revealed significant connections between IC measures and firm performance. First, the VAIC, as an aggregate measure, significantly impacts both firm profitability and value. When the VAIC is decomposed into its breakdowns, it is revealed that structural capital efficiency substantially affects firm value, and capital employed efficiency has the same function for firm profitability. In contrast to the prevalent belief in the area, human capital efficiency’s impact is found to be less important than the others. Nonlinearities are also detected in the relationships. Originality/value As ML tools are most recently introduced to the IC literature, only a few studies have used them to expand the current knowledge. However, none of these studies investigated the role of IC as a determinant of firm performance. The present study fills this gap in the literature by investigating the effect of IC efficiency on firm performance using supervised ML methods. It also provides a novel approach by comparing the estimation results of three LASSO estimators. To the best of the author’s knowledge, this is the first study that has used LASSO in IC research.

Minimax rates of convergence for sliced inverse regression with differential privacy

Sliced inverse regression (SIR) is a highly efficient paradigm used for the purpose of dimension reduction by replacing high-dimensional covariates with a limited number of linear combinations. This paper focuses on the implementation of the classical SIR approach integrated with a Gaussian differential privacy mechanism to estimate the central space while preserving privacy. We illustrate the tradeoff between statistical accuracy and privacy in sufficient dimension reduction problems under both the classical low- dimensional and modern high-dimensional settings. Additionally, we achieve the minimax rate of the proposed estimator with Gaussian differential privacy constraint and illustrate that this rate is also optimal for multiple index models with bounded dimension of the central space. Extensive numerical studies on synthetic data sets are conducted to assess the effectiveness of the proposed technique in finite sample scenarios, and a real data analysis is presented to showcase its practical application.