Arbitrary Covariance Research Articles

Direct Discrepancy Dynamic Fit Index Cutoffs for Arbitrary Covariance Structure Models

Despite the popularity of traditional fit index cutoffs like RMSEA ≤ .06 and CFI ≥ .95, several studies have noted issues with overgeneralizing traditional cutoffs. Computational methods have been proposed to avoid overgeneralization by deriving cutoffs specifically tailored to the characteristics of the model being evaluated. Simulations show favorable performance of these methods; however, these methods support a narrow set of scenarios (e.g., certain models or response scales) and the interpretation of cutoffs is not always standardized, which affects empirical researchers’ ability to confidently and broadly adopt these methods to evaluate model fit. In this paper, we propose an extension to one recently developed computational method—dynamic fit index cutoffs—that (a) permits application to any covariance structure model (e.g., CFA, mediation, bifactor), (b) standardizes interpretation of cutoffs across any covariance structure model, and (c) supports normal, nonnormal, categorical, and missing data. Software is provided to facilitate implementation of the method.

Phase transition and higher order analysis of Lq regularization under dependence.

We study the problem of estimating a [Formula: see text]-sparse signal [Formula: see text] from a set of noisy observations [Formula: see text] under the model [Formula: see text], where [Formula: see text] is the measurement matrix the row of which is drawn from distribution [Formula: see text]. We consider the class of [Formula: see text]-regularized least squares (LQLS) given by the formulation [Formula: see text], where [Formula: see text] [Formula: see text] denotes the [Formula: see text]-norm. In the setting [Formula: see text] with fixed [Formula: see text] and [Formula: see text], we derive the asymptotic risk of [Formula: see text] for arbitrary covariance matrix [Formula: see text] that generalizes the existing results for standard Gaussian design, i.e. [Formula: see text]. The results were derived from the non-rigorous replica method. We perform a higher-order analysis for LQLS in the small-error regime in which the first dominant term can be used to determine the phase transition behavior of LQLS. Our results show that the first dominant term does not depend on the covariance structure of [Formula: see text] in the cases [Formula: see text] and [Formula: see text] which indicates that the correlations among predictors only affect the phase transition curve in the case [Formula: see text] a.k.a. LASSO. To study the influence of the covariance structure of [Formula: see text] on the performance of LQLS in the cases [Formula: see text] and [Formula: see text], we derive the explicit formulas for the second dominant term in the expansion of the asymptotic risk in terms of small error. Extensive computational experiments confirm that our analytical predictions are consistent with numerical results.

Out-of-sample error estimation for M-estimators with convex penalty

Abstract A generic out-of-sample error estimate is proposed for $M$-estimators regularized with a convex penalty in high-dimensional linear regression where $(\boldsymbol{X},\boldsymbol{y})$ is observed and the dimension $p$ and sample size $n$ are of the same order. The out-of-sample error estimate enjoys a relative error of order $n^{-1/2}$ in a linear model with Gaussian covariates and independent noise, either non-asymptotically when $p/n\le \gamma $ or asymptotically in the high-dimensional asymptotic regime $p/n\to \gamma ^{\prime}\in (0,\infty )$. General differentiable loss functions $\rho $ are allowed provided that the derivative of the loss is 1-Lipschitz; this includes the least-squares loss as well as robust losses such as the Huber loss and its smoothed versions. The validity of the out-of-sample error estimate holds either under a strong convexity assumption, or for the L1-penalized Huber M-estimator and the Lasso under a sparsity assumption and a bound on the number of contaminated observations. For the square loss and in the absence of corruption in the response, the results additionally yield $n^{-1/2}$-consistent estimates of the noise variance and of the generalization error. This generalizes, to arbitrary convex penalty and arbitrary covariance, estimates that were previously known for the Lasso.

Multiple multi-sample testing under arbitrary covariance dependency.

Modern high-throughput biomedical devices routinely produce data on a large scale, and the analysis of high-dimensional datasets has become commonplace in biomedical studies. However, given thousands or tens of thousands of measured variables in these datasets, extracting meaningful features poses a challenge. In this article, we propose a procedure to evaluate the strength of the associations between a nominal (categorical) response variable and multiple features simultaneously. Specifically, we propose a framework of large-scale multiple testing under arbitrary correlation dependency among test statistics. First, marginal multinomial regressions are performed for each feature individually. Second, we use an approach of multiple marginal models for each baseline-category pair to establish asymptotic joint normality of the stacked vector of the marginal multinomial regression coefficients. Third, we estimate the (limiting) covariance matrix between the estimated coefficients from all marginal models. Finally, our approach approximates the realized false discovery proportion of a thresholding procedure for the marginal p-values for each baseline-category logit pair. The proposed approach offers a sensible trade-off between the expected numbers of true and false findings. Furthermore, we demonstrate a practical application of the method on hyperspectral imaging data. This dataset is obtained by a matrix-assisted laser desorption/ionization (MALDI) instrument. MALDI demonstrates tremendous potential for clinical diagnosis, particularly for cancer research. In our application, the nominal response categories represent cancer (sub-)types.

Estimating the proportion of signal variables under arbitrary covariance dependence

Estimating the proportion of signals hidden in a large number of noise variables is a pervasive objective in scientific research. In this paper, we consider realistic, yet theoretically challenging scenarios with arbitrary covariance dependencies between variables. We quantify the overall level of covariance dependence using mean absolute correlation (MAC), and investigate the performance of a family of estimators across the full range of MAC values. We explore the joint effect of MAC dependence, signal sparsity, and signal intensity on estimator performance, and find that no single estimator in the family performs optimally across all MAC dependence levels. Based on this theoretical insight, we propose a new estimator that is better suited to arbitrary covariance dependencies. Our method compares favorably to several existing methods in a variety of finite-sample settings, including those with strong or weak covariance dependencies and real dependence structures from genetic association studies.

Multiple two-sample testing under arbitrary covariance dependency with an application in imaging mass spectrometry.

Large-scale hypothesis testing has become a ubiquitous problem in high-dimensional statistical inference, with broad applications in various scientific disciplines. One relevant application is constituted by imaging mass spectrometry (IMS) association studies, where a large number of tests are performed simultaneously in order to identify molecular masses that are associated with a particular phenotype, for example, a cancer subtype. Mass spectra obtained from matrix-assisted laser desorption/ionization (MALDI) experiments are dependent, when considered as statistical quantities. False discovery proportion (FDP) estimation and control under arbitrary dependency structure among test statistics is an active topic in modern multiple testing research. In this context, we are concerned with the evaluation of associations between the binary outcome variable (describing the phenotype) and multiple predictors derived from MALDI measurements. We propose an inference procedure in which the correlation matrix of the test statistics is utilized. The approach is based on multiple marginal models. Specifically, we fit a marginal logistic regression model for each predictor individually. Asymptotic joint normality of the stacked vector of the marginal regression coefficients is established under standard regularity assumptions, and their (limiting) correlation matrix is estimated. The proposed method extracts common factors from the resulting empirical correlation matrix. Finally, we estimate the realized FDP of a thresholding procedure for the marginal p-values. We demonstrate a practical application of the proposed workflow to MALDI IMS data in an oncologicalcontext.

The asymptotic distribution of the MLE in high-dimensional logistic models: Arbitrary covariance

We study the distribution of the maximum likelihood estimate (MLE) in high-dimensional logistic models, extending the recent results from Sur (2019) to the case where the Gaussian covariates may have an arbitrary covariance structure. We prove that in the limit of large problems holding the ratio between the number $p$ of covariates and the sample size $n$ constant, every finite list of MLE coordinates follows a multivariate normal distribution. Concretely, the $j$th coordinate $\hat {\beta}_j$ of the MLE is asymptotically normally distributed with mean $\alpha_\star \beta_j$ and standard deviation $\sigma_\star/\tau_j$; here, $\beta_j$ is the value of the true regression coefficient, and $\tau_j$ the standard deviation of the $j$th predictor conditional on all the others. The numerical parameters $\alpha_\star > 1$ and $\sigma_\star$ only depend upon the problem dimensionality $p/n$ and the overall signal strength, and can be accurately estimated. Our results imply that the MLE's magnitude is biased upwards and that the MLE's standard deviation is greater than that predicted by classical theory. We present a series of experiments on simulated and real data showing excellent agreement with the theory.

Rotation Estimation Based on Anisotropic Angular Radon Spectrum

In this paper, we present the anisotropic Angular Radon Spectrum (ARS), a novel feature for global estimation of rotation in a two dimension space. ARS effectively describes collinearity of points and has the properties of translation-invariance and shift-rotation. We derive the analytical expression of ARS for Gaussian Mixture Models (GMM) representing point clouds where the Gaussian kernels have arbitrary covariances. Furthermore, we developed a preliminary procedure for simplification of GMM suitable for efficient computation of ARS. Rotation between point clouds is estimated by searching of maximum of correlation between their spectra. Correlation is efficiently computed from Fourier series expansion of ARS. Experiments on datasets of distorted object shapes, laser scans and on robotic mapping datasets assess the accuracy and robustness to noise in global rotation estimation.

Model‐based Bayesian inference under computer assisted balance‐improving designs

To improve covariate balance over a complete randomization, a number of methods have been proposed recently to utilize modern computational capabilities to find allocations with balance in observed covariates. Asymptotic inference on treatment effects based on these designs is more complicated than that under complete randomization, and this is why Fisher randomization tests often are suggested. This article suggests model-based Bayesian inference as a general method of inference in these designs, which can deal with complications such as arbitrary covariate balancing criteria and complex estimands. As an illustration, we focus on the case when the outcome is linearly related to the covariates and the estimand of interest is the Sample Average Treatment Effect (SATE). We use a large Monte Carlo simulation to compare the finite sample performance of the model-based Bayesian inference with that of two previous methods which are valid for asymptotic inference of SATE under Mahalanobis distance based rerandomization. We find that for experiments with small to moderate sample sizes, Bayesian inference is to be preferred to the previous methods. As a byproduct, we also find that regression adjustment together with small sample adjusted estimators of standard errors perform better than the previous methods.

Effective persistency evaluation via exact excursion distributions for random processes and fields

Finding the probability that a stochastic system stays in a certain region of its state space over a specified time—a long-standing problem both in computational physics and in applied and theoretical mathematics—is approached through the extended and multivariate Rice formula. In principle, it applies to any smooth process multivariate both in argument and in value given that efficient numerical implementations of the high-dimensional integration are available. The computational method offers an exact integral representation yielding remarkably accurate results and provides an alternative method of computing persistency probability and exponent for a physical system. It can be viewed as an implementation of path integration for a smooth Gaussian process with an arbitrary covariance. Its high accuracy is due to efficient computation of expectations with respect to high-dimensional nearly singular Gaussian distributions. For Gaussian processes, the computations are effective and more precise than those based on the Rice series expansions and the independent interval approximation. For the benchmark diffusion process, it produces the persistency exponent that is essentially the same as the recently obtained analytical value and surpasses accuracy, interpretability as well as control of the error, previous methods including the independent or Markovian approximation. The method solves the two-step excursion dependence for a stationary differentiable Gaussian process, in both theoretical and numerical sense. The solution is based on exact expressions for the probability density for one and two successive excursion lengths. The numerical routine RIND computes the densities using recent advances in scientific computing and is easily accessible for a general covariance function, via a simple numerical interface. The work offers also analytical results that explain the effectiveness of the implemented methodology and elaborates its utilization for non-Gaussian processes.

An approximate randomization test for the high-dimensional two-sample Behrens–Fisher problem under arbitrary covariances

Summary This paper is concerned with the problem of comparing the population means of two groups of independent observations. An approximate randomization test procedure based on the test statistic of Chen & Qin (2010) is proposed. The asymptotic behaviour of the test statistic, as well as the randomized statistic, is studied under weak conditions. In our theoretical framework, observations are not assumed to be identically distributed even within groups. No condition on the eigenstructure of the covariance matrices is imposed. Furthermore, the sample sizes of the two groups are allowed to be unbalanced. Under general conditions, all possible asymptotic distributions of the test statistic are obtained. We derive the asymptotic level and local power of the approximate randomization test procedure. Our theoretical results show that the proposed test procedure can adapt to all possible asymptotic distributions of the test statistic, always has the correct test level asymptotically and has good power behaviour. Our numerical experiments show that the proposed test procedure has favourable performance compared with several alternative test procedures.

A new perspective on the simulation of cross-correlated random fields

Cross-correlated random fields are widely used to model multiple uncertain parameters and/or phenomena with inherent spatial/temporal variability in numerous engineering systems. The effective representation of such fields is therefore the key element in the stochastic simulation, reliability analysis and safety assessment of engineering problems with mutual correlations. However, the simulation of such fields is generally not straightforward given the complexity of correlation structure. In this paper, we develop a unified framework for simulating non-Gaussian and non-stationary cross-correlated random fields that have been specified by their correlation structure and marginal cumulative distribution functions. Our method firstly represents the cross-correlated random fields by means of a new general stochastic expansion, in which the fields are expanded in terms of a set of deterministic functions with corresponding random variables. A finite element discretization scheme is then developed to further approximate the fields, so that the sets of deterministic functions reflecting the cross-covariance structure can be straightforwardly determined from the spectral decomposition of the resulting discretized fields. For non-Gaussian random fields, an iterative mapping procedure is developed to generate random variables to fit non-Gaussian marginal distribution of the fields. By virtue of the remarkable property of the presented stochastic expansion, i.e., various random fields share an identical set of random variables, the framework we develop is conceptually simple for simulating non-Gaussian cross-correlated fields with arbitrary covariance functions, which need not be stationary. In particular, the developed method is further generalized to a consistent framework for the simulation of multi-dimensional random fields. Five illustrative examples, including a spatially varying non-Gaussian and nonstationary seismic ground motions, are used to demonstrate the application of the developed method.

Recovering Data Permutations From Noisy Observations: The Linear Regime

This article considers a noisy data structure recovery problem. The goal is to investigate the following question: given a noisy observation of a permuted data set, according to which permutation was the original data sorted? The focus is on scenarios where data is generated according to an isotropic Gaussian distribution, and the noise is additive Gaussian with an arbitrary covariance matrix. This problem is posed within a hypothesis testing framework. The objective is to study the linear regime in which the optimal decoder has a polynomial complexity in the data size, and it declares the permutation by simply computing a permutation-independent linear function of the noisy observations. The main result of this article is a complete characterization of the linear regime in terms of the noise covariance matrix. Specifically, it is shown that this matrix must have a very flat spectrum with at most three distinct eigenvalues to induce the linear regime. Several practically relevant implications of this result are discussed, and the error probability incurred by the decision criterion in the linear regime is also characterized. A core technical component consists of using linear algebraic and geometric tools, such as Steiner symmetrization.

Large scale analysis of generalization error in learning using margin based classification methods

Large-margin classifiers are popular methods for classification. We derive the asymptotic expression for the generalization error of a family of large-margin classifiers in the limit of both sample size n and dimension p going to ∞ with fixed ratio α = n/p. This family covers a broad range of commonly used classifiers including support vector machine, distance weighted discrimination, and penalized logistic regression. Our result can be used to establish the phase transition boundary for the separability of two classes. We assume that the data are generated from a single multivariate Gaussian distribution with arbitrary covariance structure. We explore two special choices for the covariance matrix: spiked population model and two layer neural networks with random first layer weights. The method we used for deriving the closed-form expression is from statistical physics known as the replica method. Our asymptotic results match simulations already when n, p are of the order of a few hundreds. For two layer neural networks, we reproduce the recently observed ‘double descent’ phenomenology for several classification models. We also discuss some statistical insights that can be drawn from these analysis.

Multiscale functional inequalities in probability: Constructive approach

Consider an ergodic stationary random field A on the ambient space ℝ d . In order to establish concentration properties for nonlinear functions Z(A), it is standard to appeal to functional inequalities like Poincaré or logarithmic Sobolev inequalities in the probability space. These inequalities are however only known to hold for a restricted class of laws (product measures, Gaussian measures with integrable covariance, or more general Gibbs measures with nicely behaved Hamiltonians). In this contribution, we introduce variants of these inequalities, which we refer to as multiscale functional inequalities and which still imply fine concentration properties, and we develop a constructive approach to such inequalities. We consider random fields that can be viewed as transformations of a product structure, for which the question is reduced to devising approximate chain rules for nonlinear random changes of variables. This approach allows us to cover most examples of random fields arising in the modelling of heterogeneous materials in the applied sciences, including Gaussian fields with arbitrary covariance function, Poisson random inclusions with (unbounded) random radii, random parking and Matérn-type processes, as well as Poisson random tessellations. The obtained multiscale functional inequalities, which we primarily develop here in view of their application to concentration and to quantitative stochastic homogenization, are of independent interest.

Using singular spectrum analysis for inference on seasonal time series with seasonal unit roots

The problem of optimal linear filtering, smoothing and trend extraction for m-period differences of processes with a unit root is studied. Such processes arise naturally in economics and finance, in the form of rates of change (price inflation, economic growth, financial returns) and finding an appropriate smoother is thus of immediate practical interest. The filter and resulting smoother are based on the methodology of singular spectrum analysis (SSA). An explicit representation for the asymptotic decomposition of the covariance matrix is obtained. The structure of the impulse and frequency response functions indicates that the optimal filter has a 'permanent' and a 'transitory component', with the corresponding smoother being the sum of two such components. Moreover, a particular form for the extrapolation coefficients that can be used in out-of-sample prediction is proposed. In addition, an explicit representation for the filtering weights in the context of SSA for an arbitrary covariance matrix is derived. This result allows one to examine the specific effects of smoothing in any situation. The theoretical results are illustrated using different datasets, namely US inflation and real GDP growth.

Using singular spectrum analysis for inference on seasonal time series with seasonal unit roots

The problem of optimal linear filtering, smoothing and trend extraction for m-period differences of processes with a unit root is studied. Such processes arise naturally in economics and finance, in the form of rates of change (price inflation, economic growth, financial returns) and finding an appropriate smoother is thus of immediate practical interest. The filter and resulting smoother are based on the methodology of singular spectrum analysis (SSA). An explicit representation for the asymptotic decomposition of the covariance matrix is obtained. The structure of the impulse and frequency response functions indicates that the optimal filter has a 'permanent' and a 'transitory component', with the corresponding smoother being the sum of two such components. Moreover, a particular form for the extrapolation coefficients that can be used in out-of-sample prediction is proposed. In addition, an explicit representation for the filtering weights in the context of SSA for an arbitrary covariance matrix is derived. This result allows one to examine the specific effects of smoothing in any situation. The theoretical results are illustrated using different datasets, namely US inflation and real GDP growth.

Optimal clustering with missing values

BackgroundMissing values frequently arise in modern biomedical studies due to various reasons, including missing tests or complex profiling technologies for different omics measurements. Missing values can complicate the application of clustering algorithms, whose goals are to group points based on some similarity criterion. A common practice for dealing with missing values in the context of clustering is to first impute the missing values, and then apply the clustering algorithm on the completed data.ResultsWe consider missing values in the context of optimal clustering, which finds an optimal clustering operator with reference to an underlying random labeled point process (RLPP). We show how the missing-value problem fits neatly into the overall framework of optimal clustering by incorporating the missing value mechanism into the random labeled point process and then marginalizing out the missing-value process. In particular, we demonstrate the proposed framework for the Gaussian model with arbitrary covariance structures. Comprehensive experimental studies on both synthetic and real-world RNA-seq data show the superior performance of the proposed optimal clustering with missing values when compared to various clustering approaches.ConclusionOptimal clustering with missing values obviates the need for imputation-based pre-processing of the data, while at the same time possessing smaller clustering errors.

False Discovery Rate Control in Cancer Biomarker Selection Using Knockoffs.

The discovery of biomarkers that are informative for cancer risk assessment, diagnosis, prognosis and treatment predictions is crucial. Recent advances in high-throughput genomics make it plausible to select biomarkers from the vast number of human genes in an unbiased manner. Yet, control of false discoveries is challenging given the large number of genes versus the relatively small number of patients in a typical cancer study. To ensure that most of the discoveries are true, we employ a knockoff procedure to control false discoveries. Our method is general and flexible, accommodating arbitrary covariate distributions, linear and nonlinear associations, and survival models. In simulations, our method compares favorably to the alternatives; its utility of identifying important genes in real clinical applications is demonstrated by the identification of seven genes associated with Breslow thickness in skin cutaneous melanoma patients.

Efficient Computation of Multivariate Rayleigh and Exponential Distributions

We propose an efficient approach for the computation of cumulative distribution functions of ${N}$ correlated Rayleigh or exponential random variables (RVs) for arbitrary covariance matrices, which arise in the design and analysis of many wireless systems. Compared to the approaches in the literature, it employs a fast and accurate randomized quasi-Monte Carlo method that markedly reduces the computational complexity by several orders of magnitude as ${N}$ or the correlation among the RVs increases. Numerical results show that an order of magnitude larger values of ${N}$ can now be computed for. Its application to the performance analysis of selection combining is also shown.