Covariance Estimation Research Articles

Challenges for Inline Observation Error Estimation in the Presence of Misspecified Background Uncertainty

Abstract For data assimilation to provide faithful state estimates for dynamical models, specifications of observation uncertainty need to be as accurate as possible. Innovation-based methods based on Desroziers diagnostics, are commonly used to estimate observation uncertainty, but such methods can depend greatly on the prescribed background uncertainty. For ensemble data assimilation, this uncertainty comes from statistics calculated from ensemble forecasts, which require inflation and localization to address under sampling. In this work, we use an ensemble Kalman filter (EnKF) with a low-dimensional Lorenz model to investigate the interplay between the Desroziers method and inflation. Two inflation techniques are used for this purpose: 1) a rigorously tuned fixed multiplicative scheme and 2) an adaptive state-space scheme. We document how inaccuracies in observation uncertainty affect errors in EnKF posteriors and study the combined impacts of misspecified initial observation uncertainty, sampling error, and model error on Desroziers estimates. We find that whether observation uncertainty is over- or underestimated greatly affects the stability of data assimilation and the accuracy of Desroziers estimates and that preference should be given to initial overestimates. Inline estimates of Desroziers tend to remove the dependence between ensemble spread–skill and the initially prescribed observation error. In addition, we find that the inclusion of model error introduces spurious correlations in observation uncertainty estimates. Further, we note that the adaptive inflation scheme is less robust than fixed inflation at mitigating multiple sources of error. Last, sampling error strongly exacerbates existing sources of error and greatly degrades EnKF estimates, which translates into biased Desroziers estimates of observation error covariance. Significance Statement To generate accurate predictions of various components of the Earth system, numerical models require an accurate specification of state variables at our current time. This step adopts a probabilistic consideration of our current state estimate versus information provided from environmental measurements of the true state. Various strategies exist for estimating uncertainty in observations within this framework, but are sensitive to a host of assumptions, which are investigated in this study.

Correcting for informative sampling in spatial covariance estimation and kriging predictions

Correcting for informative sampling in spatial covariance estimation and kriging predictions

Spatial Channel Covariance Estimation and Two-Timescale Beamforming for IRS-Assisted Millimeter Wave Systems

We consider the problem of spatial channel covariance matrix (CCM) estimation for intelligent reflecting surface (IRS)-assisted millimeter wave (mmWave) communication systems. Spatial CCM is essential for two-timescale beamforming in IRS-assisted systems; however, estimating the spatial CCM is challenging due to the passive nature of reflecting elements and the large size of the CCM resulting from massive reflecting elements of the IRS. In this paper, we propose a CCM estimation method by exploiting the low-rankness as well as the positive semi-definite (PSD) 3-level Toeplitz structure of the CCM. Estimation of the CCM is formulated as a semidefinite programming (SDP) problem and an alternating direction method of multipliers (ADMM) algorithm is developed. Our analysis shows that the proposed method is theoretically guaranteed to attain a reliable CCM estimate with a sample complexity much smaller than the dimension of the CCM. Thus the proposed method can help achieve a significant training overhead reduction. Simulation results are presented to illustrate the effectiveness of our proposed method and the performance of two-timescale beamforming scheme based on the estimated CCM.

Cell-wise robust covariance estimation for compositions, with application to geochemical data

Cell-wise outliers are outliers in single entries of a compositional data matrix, and they can lead to a certain bias in the statistical analysis. Traditional row-wise robust methods downweight outlying observations for the estimation, independent of how many or which cells of an observation are contaminated. Cell-wise robustness still makes use of the information contained in non-contaminated cells. Here, cell-wise robustness is used for the estimation of the variation and the covariance matrix. For higher dimensional data also a regularized estimator is introduced. The advantages of the cell-wise robust estimators are demonstrated in simulation experiments and in a geochemistry application in the context of clustering and principal component analysis.

Estimation and consequences of direct-maternal genetic and environmental covariances in models for genetic evaluation in broilers

BackgroundMaternal effects influence juvenile traits such as body weight and early growth in broilers. Ignoring significant maternal effects leads to reduced accuracy and inflated predicted breeding values. Including genetic and environmental direct-maternal covariances into prediction models in broilers can increase the accuracy and limit inflation of predicted breeding values better than simply adding maternal effects to the model. To test this hypothesis, we applied a model accounting for direct-maternal genetic covariance and direct-maternal environmental covariance to estimate breeding values.ResultsThis model, and simplified versions of it, were tested using simulated broiler populations and then was applied to a large broiler population for validation. The real population analyzed consisted of a commercial line of broilers, for which body weight at a common slaughter age was recorded for 41 selection rounds. The direct-maternal genetic covariance was negative whereas the direct-maternal environmental covariance was positive. Simulated populations were created to mimic the real population. The predictive ability of the models was assessed by cross-validation, where the validation birds were all from the last five selection rounds. Accuracy of prediction was defined as the correlation between the predicted breeding values estimated without the phenotypic records of the validation population and a predictor. The predictors were the breeding values estimated using all the phenotypic information and the phenotypes corrected for the fixed effects, and for the simulated data, the true breeding values. In the real data, adding the environmental covariance, with or without also adding the genetic covariance, increased the accuracy, or reduced deflation of breeding values compared with a model not including dam–offspring covariance. Nevertheless, in the simulated data, reduction in the inflation of breeding values was possible and was associated with a gain in accuracy of up to 6% compared with a model not including both forms of direct-maternal covariance.ConclusionsIn this paper, we propose a simple approach to estimate the environmental direct-maternal covariance using standard software for REML analysis. The genetic covariance between dam and offspring was negative whereas the corresponding environmental covariance was positive. Considering both covariances in models for genetic evaluation increased the accuracy of predicted breeding values.

Core shrinkage covariance estimation for matrix-variate data

Abstract A separable covariance model can describe the among-row and among-column correlations of a random matrix and permits likelihood-based inference with a very small sample size. However, if the assumption of separability is not met, data analysis with a separable model may misrepresent important dependence patterns in the data. As a compromise between separable and unstructured covariance estimation, we decompose a covariance matrix into a separable component and a complementary ‘core’ covariance matrix. This decomposition defines a new covariance matrix decomposition that makes use of the parsimony and interpretability of a separable covariance model, yet fully describes covariance matrices that are non-separable. This decomposition motivates a new type of shrinkage estimator, obtained by appropriately shrinking the core of the sample covariance matrix, that adapts to the degree of separability of the population covariance matrix.

Large covariance estimation using a factor model with common and group‐specific factors

AbstractThis paper proposes a new approach to estimate large covariance matrices using multilevel factor models. In order to further improve the efficiency of the principal orthogonal complement thresholding estimator (PEOT) and the proposed estimators, the generalized least squares (GLS) method is employed to refine the estimation of the factors. A novel approach to identify number of the factors is proposed for facilitating our estimation procedure. We prove the consistency of the covariance matrix estimators and the estimators for number of the factors. Our Monte Carlo simulations show that the proposed estimators have superior properties in finite samples for all different designs, and the efficiency can be improved significantly by using GLS. Finally, we apply our estimators to a dataset consisting of weekly returns of three major stock indexes constituents, and the results suggest that the proposed methods can improve the out‐of‐sample performances of portfolio optimization.

Validation of semi-analytical, semi-empirical covariance matrices for two-point correlation function for early DESI data

ABSTRACT We present an extended validation of semi-analytical, semi-empirical covariance matrices for the two-point correlation function (2PCF) on simulated catalogs representative of luminous red galaxies (LRGs) data collected during the initial 2 months of operations of the Stage-IV ground-based Dark Energy Spectroscopic Instrument (DESI). We run the pipeline on multiple effective Zel’dovich (EZ) mock galaxy catalogs with the corresponding cuts applied and compare the results with the mock sample covariance to assess the accuracy and its fluctuations. We propose an extension of the previously developed formalism for catalogs processed with standard reconstruction algorithms. We consider methods for comparing covariance matrices in detail, highlighting their interpretation and statistical properties caused by sample variance, in particular, non-trivial expectation values of certain metrics even when the external covariance estimate is perfect. With improved mocks and validation techniques, we confirm a good agreement between our predictions and sample covariance. This allows one to generate covariance matrices for comparable data sets without the need to create numerous mock galaxy catalogs with matching clustering, only requiring 2PCF measurements from the data itself. The code used in this paper is publicly available at https://github.com/oliverphilcox/RascalC.

Post-processed posteriors for sparse covariances

We consider Bayesian inference of sparse covariance matrices and propose a post-processed posterior. This method consists of two steps. In the first step, posterior samples are obtained from the conjugate inverse-Wishart posterior without considering the sparse structural assumption. The posterior samples are transformed in the second step to satisfy the sparse structural assumption through a generalized thresholding function. This non-traditional Bayesian procedure is justified by showing that the post-processed posterior attains the optimal minimax rates under the spectral norm loss in high-dimensional settings. We also propose the post-processed posterior for contaminated data and apply it to the estimation of the sparse idiosyncratic covariance of the approximate factor model. The advantages of our method are demonstrated via a simulation study and a real data analysis with S&P 500 data.

Improved process representation of leaf phenology significantly shifts climate sensitivity of ecosystem carbon balance

Abstract. Terrestrial carbon cycle models are routinely used to determine the response of the land carbon sink under expected future climate change, yet these predictions remain highly uncertain. Increasing the realism of processes in these models may help with predictive skill, but any such addition should be confronted with observations and evaluated in the context of the aggregate behavior of the carbon cycle. Here, two formulations for leaf area index (LAI) phenology are coupled to the same terrestrial biosphere model: one is climate agnostic, and the other incorporates direct environmental controls on both timing and growth. Each model is calibrated simultaneously to observations of LAI, net ecosystem exchange (NEE), and biomass using the CARbon DAta-MOdel fraMework (CARDAMOM) and validated against withheld data, including eddy covariance estimates of gross primary productivity (GPP) and ecosystem respiration (Re) across six ecosystems from the tropics to high latitudes. Both model formulations show similar predictive skill for LAI and NEE. However, with the addition of direct environmental controls on LAI, the integrated model explains 22 % more variability in GPP and Re and reduces biases in these fluxes by 58 % and 77 %, respectively, while also predicting more realistic annual litterfall rates due to changes in carbon allocation and turnover. We extend this analysis to evaluate the inferred climate sensitivity of LAI and NEE with the new model and show that the added complexity shifts the sign, magnitude, and seasonality of NEE sensitivity to precipitation and temperature. This highlights the benefit of process complexity when inferring underlying processes from Earth observations and representing the climate response of the terrestrial carbon cycle.

Characterization of direct and/or indirect genetic associations for multiple traits in longitudinal studies of disease progression.

When quantitative longitudinal traits are risk factors for disease progression and subject to random biological variation, joint model analysis of time-to-event and longitudinal traits can effectively identify direct and/or indirect genetic association of single nucleotide polymorphisms (SNPs) with time-to-event. We present a joint model that integrates: (1) a multivariate linear mixed model describing trajectories of multiple longitudinal traits as a function of time, SNP effects, and subject-specific random effects and (2) a frailty Cox survival model that depends on SNPs, longitudinal trajectory effects, and subject-specific frailty accounting for dependence among multiple time-to-event traits. Motivated by complex genetic architecture of type 1 diabetes complications (T1DC) observed in the Diabetes Control and Complications Trial (DCCT), we implement a 2-stage approach to inference with bootstrap joint covariance estimation and develop a hypothesis testing procedure to classify direct and/or indirect SNP association with each time-to-event trait. By realistic simulation study, we show that joint modeling of 2 time-to-T1DC (retinopathy and nephropathy) and 2 longitudinal risk factors (HbA1c and systolic blood pressure) reduces estimation bias in genetic effects and improves classification accuracy of direct and/or indirect SNP associations, compared to methods that ignore within-subject risk factor variability and dependence among longitudinal and time-to-event traits. Through DCCT data analysis, we demonstrate feasibility for candidate SNP modeling and quantify effects of sample size and Winner's curse bias on classification for 2 SNPs identified as having indirect associations with time-to-T1DC traits. Joint analysis of multiple longitudinal and multiple time-to-event traits provides insight into complex traits architecture.

Bayesian sparse seemingly unrelated regressions model with variable selection and covariance estimation via the horseshoe+

Bayesian sparse seemingly unrelated regressions model with variable selection and covariance estimation via the horseshoe+

Self-initiated expatriates motivation and basic psychological needs – a generalization and extension replication

PurposeThis research aims to replicate the work of Oberholster et al. (2013) on expatriation motivation through a generalization and extension replication. Additionally, it aims to contribute to the Self-Initiated Expatriates (SIEs) literature by studying the Basic Psychological Needs (BPNs) of SIEs as proposed by the Self-Determination Theory (SDT).Design/methodology/approachThis research used Latent Class Analysis (LCA) with covariate estimation to create five clusters based on survey data from 179 SIEs. Additionally, the replication procedure followed recommendations by Dau et al. (2022) to produce a constructive replication.FindingsBesides validating clusters in the original study, one new cluster was found. Furthermore, identifying the BPNs of the clusters helped in understanding the mechanism that motivates them.Research limitations/implicationsBy studying the BPNs as per the SDT, this research contributes to our understanding of the psychological factors that shape expatriates' motivations and experiences. This can inform the knowledge developed in the international mobility theories and the models related to SIEs' psychological well-being.Practical implicationsBuilding on the results of this research, organizations can tailor recruiting and retaining strategies to the specific BPNs of different SIEs groups. Organizations can motivate SIEs, enhance their job satisfaction and loyalty, predict their success, and support their psychological well-being.Originality/valueA generalization and extension replication value lies in testing the validity and reliability of previous findings in new contexts. The originality in this research stems from its utilization of the BPNs from SDT to explain SIEs' motivation and evaluate their psychological well-being.

Snow–vegetation–atmosphere interactions in alpine tundra

Abstract. The interannual variability of snow cover in alpine areas is increasing, which may affect the tightly coupled cycles of carbon and water through snow–vegetation–atmosphere interactions across a range of spatio-temporal scales. To explore the role of snow cover for the land–atmosphere exchange of CO2 and water vapor in alpine tundra ecosystems, we combined 3 years (2019–2021) of continuous eddy covariance flux measurements of the net ecosystem exchange of CO2 (NEE) and evapotranspiration (ET) from the Finse site in alpine Norway (1210 m a.s.l.) with a ground-based ecosystem-type classification and satellite imagery from Sentinel-2, Landsat 8, and MODIS. While the snow conditions in 2019 and 2021 can be described as site typical, 2020 features an extreme snow accumulation associated with a strong negative phase of the Scandinavian pattern of the synoptic atmospheric circulation during spring. This extreme snow accumulation caused a 1-month delay in melt-out date, which falls in the 92nd percentile in the distribution of yearly melt-out dates in the period 2001–2021. The melt-out dates follow a consistent fine-scale spatial relationship with ecosystem types across years. Mountain and lichen heathlands melt out more heterogeneously than fens and flood plains, while late snowbeds melt out up to 1 month later than the other ecosystem types. While the summertime average normalized difference vegetation index (NDVI) was reduced considerably during the extreme-snow year 2020, it reached the same maximum as in the other years for all but one of the ecosystem types (late snowbeds), indicating that the delayed onset of vegetation growth is compensated to the same maximum productivity. Eddy covariance estimates of NEE and ET are gap-filled separately for two wind sectors using a random forest regression model to account for complex and nonlinear ecohydrological interactions. While the two wind sectors differ markedly in vegetation composition and flux magnitudes, their flux response is controlled by the same drivers as estimated by the predictor importance of the random forest model, as well as by the high correlation of flux magnitudes (correlation coefficient r=0.92 for NEE and r=0.89 for ET) between both areas. The 1-month delay of the start of the snow-free season in 2020 reduced the total annual ET by 50 % compared to 2019 and 2021 and reduced the growing-season carbon assimilation to turn the ecosystem from a moderate annual carbon sink (−31 to −6 gC m−2 yr−1) to a source (34 to 20 gC m−2 yr−1). These results underpin the strong dependence of ecosystem structure and functioning on snow dynamics, whose anomalies can result in important ecological extreme events for alpine ecosystems.

Estimating the Population Mean in Stratified Random Sampling Using Combined Regression with the Presence of Outliers

In this research, the covariance estimates were used to estimate the population mean in the stratified random sampling and combined regression estimates. were compared by employing the robust variance-covariance matrices estimates with combined regression estimates by employing the traditional variance-covariance matrices estimates when estimating the regression parameter, through the two efficiency criteria (RE) and mean squared error (MSE). We found that robust estimates significantly improved the quality of combined regression estimates by reducing the effect of outliers using robust covariance and covariance matrices estimates (MCD, MVE) when estimating the regression parameter. In addition, the results of the simulation study proved that the Minimum covariance determinant (MCD) method is highly efficient at all sample sizes (n=35, 75, 150, 200, 500) and then followed by the method of the smallest ellipse Minimum volume Ellipsoid (MVE) handles outliers in the dataset, where it has lower values (MSE).

A Bayesian hierarchical sparse factor model for estimating simultaneous covariance matrices for gestational outcomes in consecutive pregnancies.

Covariance estimation for multiple groups is a key feature for drawing inference from a heterogeneous population. One should seek to share information about common features in the dependence structures across the various groups. In this paper, we introduce a novel approach for estimating the covariance matrices for multiple groups using a hierarchical latent factor model that shrinks the factor loadings across groups toward a global value. Using a sparse spike and slab model on these loading coefficients allows for a sparse formulation of our model. Parameter estimation is accomplished through a Markov chain Monte Carlo scheme, and a model selection approach is used to select the number of factors to use. We validate our model through extensive simulation studies. Finally, we apply our methodology to the NICHD Consecutive Pregnancies Study to estimate the correlations between birth weights and gestational ages of three consecutive birth within four different subgroups (underweight, normal, overweight, and obese) of women.

Realized regression with asynchronous and noisy high frequency and high dimensional data

We develop regression for high frequency data. This regression is novel in that it can be for both fixed and increasing dimension. Also, the data may have microstructure noise, and observations (trades, or quotes) can be asynchronous, (i.e., the observations do not need to be synchronized across dimensions). As is customary for high-frequency inference methods, we refer to our method as “realized” regression.In our methodology, spot beta becomes a key quantity in the nonparametric framework of high frequency econometrics. The central contribution of this paper is a feasible estimator of spot beta, which is robust to noise and asynchronicity. With the help of the spot-version of the Smoothed TSRV estimator, spot beta can be consistently estimated. There are two direct applications of the spot beta estimates in the current paper. In the first application, the integrated beta can be consistently estimated by aggregating the spot beta estimates. After a bias-correction procedure, a fixed dimension central limit theorem is established for the bias-corrected estimator, with convergence rate which may be arbitrarily close to Op(n−1/4). In the second application we assume time-varying factor structure and conditional sparsity. The spot beta matrix estimator enables the estimation of high dimensional spot covariance and precision matrices. The latter is obtained by thresholding the spot residual covariance estimates, and convergence rates derived. As an empirical application, this paper explores the hourly change in beta around earnings announcements of the S&P 100 constituents.

Reconfigured metabolism brain network in asymptomatic microtubule‐associated protein tau mutation carriers: A graph‐theoretical analysis

AbstractBackgroundStudies exploring topologic properties of the metabolic network in the presymptomatic stage of genetic frontotemporal dementia (FTD) are scarce, which may be important for understanding brain function and disease pathogenesis. This study aimed to explore FTD‐specific patterns of metabolism topology reconfiguration in asymptomatic microtubule‐associated protein tau (MAPT) mutation carriers.MethodSix asymptomatic carriers of MAPT P301L mutation were compared with 12 non‐carriers who were all from the same large family of FTD. For comparison, we included 32 behavioral‐variant FTD (bvFTD) patients and 33 unrelated healthy controls. Each participant underwent neuropsychological assessments, genetic testing, and hybrid PET/MRI scan. Voxel‐wise gray matter volume and standardized uptake value ratio were calculated and compared for structural MRI and FDG‐PET, separately. The sparse inverse covariance estimation (SICE) method was applied to topologic properties and metabolic connectomes of brain functional networks derived from 18F‐fluorodeoxyglucose PET/MRI data. Independent component analysis was used to explore metabolic connectivity in the salience (SN) and default mode networks (DMN).ResultThe mean estimated years from symptom onset was 8.33 ± 1.875 years in the mutation carrier group. The asymptomatic MAPT carriers show no significant differences in cognitive tests, gray matter volume and FDG uptake compared to non‐carriers. They also retained normal global parameters of metabolism network, but not for bvFTD patients. However, we revealed lost hubs in the ventromedial prefrontal, orbitofrontal, and anterior cingulate cortices and reconfigured hubs in the anterior insula, precuneus, and posterior cingulate cortex in asymptomatic carriers, which is consistent with the findings in bvFTD patients. Similarly, significant differences in local parameters of these nodes were observed in asymptomatic carriers. The reduction in connectivity was marked in lost hub regions during the asymptomatic stage. Functional connectivity within the SN and DMN was enhanced in asymptomatic carriers, while reduced in bvFTD patients.ConclusionOur findings showed that asymptomatic MAPT carriers were initially involved in medial prefrontal areas and actively compensated in task‐related regions. Topologic properties of the metabolic network may contribute to further investigations and monitoring of the earliest stages of FTD in individuals with genetic backgrounds.

USV Parameter Estimation: Adaptive Unscented Kalman Filter-Based Approach

In this article, a new kind of adaptive unscented Kalman filter (AUKF) is proposed to deal with the parameter estimation problem for a class of nonlinear unmanned surface vessel (USV) models with unknown statistical characteristics of process noises. Specifically, the considered parameter estimation problem is first transformed into the state estimation problem by extending eighteen parameters and three unknown inputs into augmented states for the USVs. With the help of such a transformation, the unknown inputs including disturbances and modeling errors are estimated effectively, and employed to construct the estimators such that the effect of these unknown inputs on parameter estimation can be significantly suppressed. Under the condition that the structure of the covariance matrix of the process noise is available, an adaptive law is further designed to estimate such a high-dimensional covariance matrix where the covariance estimation errors can be reduced. Finally, the proposed estimation approach is verified via performing the practical experiment as well as numerical simulations.

Robust Estimation of Position-Dependent Anisotropic Diffusivity Tensors from Stochastic Trajectories.

Materials under confinement can possess properties that deviate considerably from their bulk counterparts. Indeed, confinement makes all physical properties position-dependent and possibly anisotropic, and characterizing such spatial variations and directionality has been an intense area of focus in experimental and computational studies of confined matter. While this task is fairly straightforward for simple mechanical observables, it is far more daunting for transport properties such as diffusivity that can only be estimated from autocorrelations of mechanical observables. For instance, there are well established methods for estimating diffusivity from experimentally observed or computationally generated trajectories in bulk systems. No rigorous generalizations of such methods, however, exist for confined systems. In this work, we present two filtered covariance estimators for computing anisotropic and position-dependent diffusivity tensors and validate them by applying them to stochastic trajectories generated according to known diffusivity profiles. These estimators can accurately capture spatial variations that span over several orders of magnitude and that assume different functional forms. Our kernel-based approach is also very robust to implementation details such as the localization function and time discretization and performs significantly better than estimators that are solely based on local covariance. Moreover, the kernel function does not have to be localized and can instead belong to a dictionary of orthogonal functions. Therefore, the proposed estimator can be readily used to obtain functional estimates of diffusivity rather than a tabulated collection of pointwise estimates. Nonetheless, the susceptibility of the proposed estimators to time discretization is higher at the immediate vicinity of hard boundaries. We demonstrate this heightened susceptibility to be common among all covariance-based estimators.