Construction Of Confidence Regions Research Articles

Variance-Reduced Accelerated First-Order Methods: Central Limit Theorems and Confidence Statements

In this paper, we consider a strongly convex stochastic optimization problem and propose three classes of variable sample-size stochastic first-order methods: (i) the standard stochastic gradient descent method, (ii) its accelerated variant, and (iii) the stochastic heavy-ball method. In each scheme, the exact gradients are approximated by averaging across an increasing batch size of sampled gradients. We prove that when the sample size increases at a geometric rate, the generated estimates converge in mean to the optimal solution at an analogous geometric rate for schemes (i)–(iii). Based on this result, we provide central limit statements, whereby it is shown that the rescaled estimation errors converge in distribution to a normal distribution with the associated covariance matrix dependent on the Hessian matrix, the covariance of the gradient noise, and the step length. If the sample size increases at a polynomial rate, we show that the estimation errors decay at a corresponding polynomial rate and establish the associated central limit theorems (CLTs). Under certain conditions, we discuss how both the algorithms and the associated limit theorems may be extended to constrained and nonsmooth regimes. Finally, we provide an avenue to construct confidence regions for the optimal solution based on the established CLTs and test the theoretical findings on a stochastic parameter estimation problem. Funding: This work was supported by the Office of Naval Research [Grant N00014-22-1-2589] and Basic Energy Sciences [Grant DE-SC0023303]. The work of J. Lei was partially supported by the National Natural Science Foundation of China [Grants 72271187 and 62088101]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/moor.2021.0068 .

Estimation of Random Coefficient Autoregressive Model with Error in Covariates

Measurement error is common in many statistical problems and has received considerable attention in various regression contexts. In this study, we consider the random coefficient autoregressive model with measurement error possibly present in covariates. The least squares and weighted least squares methods are used to estimate the model parameters, and the consistency and asymptotic normality of the two kinds of estimators are proved. Furthermore, we propose an empirical likelihood method based on weighted score equations to construct confidence regions for the parameters. The simulation results show that the weighted least squares estimators are superior to the least squares estimators and that the confidence regions have good finite-sample behavior. At last, the model is applied to a real data example.

Comparison of confidence regions for quantum state tomography

Abstract The quantum state associated to an unknown experimental preparation procedure can be determined by performing quantum state tomography. If the statistical uncertainty in the data dominates over other experimental errors, then a tomographic reconstruction procedure must express this uncertainty. A rigorous way to accomplish this is via statistical confidence regions (CRs) in state space. Naturally, the size of this region decreases when increasing the number of samples, but it also depends critically on the construction method of the region. We compare recent methods for constructing CRs as well as a reference method based on a Gaussian approximation. For the comparison, we propose an operational measure with the finding, that there is a significant difference between methods, but which method is preferable can depend on the details of the state preparation scenario.

Unifying Estimation and Inference for Linear Regression with Stationary and Integrated or Near-Integrated Variables

Abstract There is a discrepancy in the limiting distributions of least-squares estimators for stationary and integrated variables. For statistical inference, it must be decided which distribution should be used in advance. This motivates us to develop a unifying inference procedure based on weighted estimation. The asymptotic distributions of the proposed estimators are developed and a random weighting bootstrap method is proposed for constructing confidence regions. The proposed method outperforms existing methods (with time constant or time-varying error variance) in simulations. We further study the predictability of asset returns in a setting where some of our state variables are endogenous.

On adaptive confidence sets for the Wasserstein distances

In the density estimation model, we investigate the problem of constructing adaptive honest confidence sets with radius measured in Wasserstein distance $W_p$, $p\geq1$, and for densities with unknown regularity measured on a Besov scale. As sampling domains, we focus on the $d-$dimensional torus $\mathbb{T}^d$, in which case $1\leq p\leq 2$, and $\mathbb{R}^d$, for which $p=1$. We identify necessary and sufficient conditions for the existence of adaptive confidence sets with diameters of the order of the regularity-dependent $W_p$-minimax estimation rate. Interestingly, it appears that the possibility of such adaptation of the diameter depends on the dimension of the underlying space. In low dimensions, $d\leq 4$, adaptation to any regularity is possible. In higher dimensions, adaptation is possible if and only if the underlying regularities belong to some interval of width at least $d/(d-4)$. This contrasts with the usual $L_p-$theory where, independently of the dimension, adaptation requires regularities to lie in a small fixed-width window. For configurations allowing these adaptive sets to exist, we explicitly construct confidence regions via the method of risk estimation, centred at adaptive estimators. Those are the first results in a statistical approach to adaptive uncertainty quantification with Wasserstein distances. Our analysis and methods extend more globally to weak losses such as Sobolev norm distances with negative smoothness indices.

Concentration inequalities for Kernel density estimators under uniform mixing

We derive non-asymptotic concentration inequalities for the uniform deviation between a multivariate density function and its non-parametric kernel density estimator in stationary and uniform mixing time series framework. We derive analogous inequalities for their (first) Wasserstein distance, as well as for the deviations between integrals of bounded functions w.r.t. them. They can be used for the construction of confidence regions, the estimation of the finite sample probabilities of decision errors, etc. We employ the concentration results to the derivation of statistical guarantees and oracle inequalities in regularized prediction problems with Lipschitz and strongly convex costs.

Inflation expectations and the Phillips curve: an encompassing framework

This paper contrasts empirically four leading models of inflation dynamics – the Accelerationist Phillips curve (APC), New Keynesian Phillips curve (NKPC), Hybrid Phillips curve (HPC) and Sticky Information Phillips curve (SIPC). We employ an encompassing Phillips curve specification that allows us to derive tests for these models within a single framework. Using the generalized method of moments (GMM) estimator, the evidence suggests that the restrictions implied by the NKPC, HPC, and SIPC are rejected for the period after the Real Plan in Brazil. Only the restrictions implied by the APC are not rejected. However, when we construct confidence regions that are robust to weak instruments, it is not possible to reject any of the Phillips curve specifications, including the NKPC.

Sample Complexity of the Sign-Perturbed Sums Identification Method: Scalar Case*

Sign-Perturbed Sum (SPS) is a powerful finite-sample system identification algorithm which can construct confidence regions for the true data generating system with exact coverage probabilities, for any finite sample size. SPS was developed in a series of papers and it has a wide range of applications, from general linear systems, even in a closed-loop setup, to nonlinear and nonparametric approaches. Although several theoretical properties of SPS were proven in the literature, the sample complexity of the method was not analysed so far. This paper aims to fill this gap and provides the first results on the sample complexity of SPS. Here, we focus on scalar linear regression problems, that is we study the behaviour of SPS confidence intervals. We provide high probability upper bounds, under three different sets of assumptions, showing that the sizes of SPS confidence intervals shrink at a geometric rate around the true parameter, if the observation noises are subgaussian. We also show that similar bounds hold for the previously proposed outer approximation of the confidence region. Finally, we present simulation experiments comparing the theoretical and the empirical convergence rates.

Ridge regression revisited: Debiasing, thresholding and bootstrap

The success of the Lasso in the era of high-dimensional data can be attributed to its conducting an implicit model selection, that is, zeroing out regression coefficients that are not significant. By contrast, classical ridge regression cannot reveal a potential sparsity of parameters, and may also introduce a large bias under the high-dimensional setting. Nevertheless, recent work on the Lasso involves debiasing and thresholding, the latter in order to further enhance the model selection. As a consequence, ridge regression may be worth another look since—after debiasing and thresholding—it may offer some advantages over the Lasso, for example, it can be easily computed using a closed-form expression. In this paper, we define a debiased and thresholded ridge regression method, and prove a consistency result and a Gaussian approximation theorem. We further introduce a wild bootstrap algorithm to construct confidence regions and perform hypothesis testing for a linear combination of parameters. In addition to estimation, we consider the problem of prediction, and present a novel, hybrid bootstrap algorithm tailored for prediction intervals. Extensive numerical simulations further show that the debiased and thresholded ridge regression has favorable finite sample performance and may be preferable in some settings.

Permutation confidence region for multiple regression and fidelity to asymptotic approximation

The permutation inference distribution (PID) introduced in Wu and Vos (2019), The Journal of Nonparametric Statistics 31(3), 722-742) is extended to the joint PID for multiple regression. The joint PID is used for both hypothesis testing and the construction of confidence regions and its computational burden is similar to that of conducting a single hypothesis test. Asymptotic normality results show that PID confidence regions are asymptotically ellipsoidal and exact. In finite samples, the PID confidence regions can be used to check the fidelity of normal approximations. In some cases, normal theory-based confidence regions may be adjusted to approach nominal confidence errors. Simulation studies and real data applications are used to evaluate inferences obtained from the PID.

Parameter estimation and uncertainty quantification using information geometry.

In this work, we: (i) review likelihood-based inference for parameter estimation and the construction of confidence regions; and (ii) explore the use of techniques from information geometry, including geodesic curves and Riemann scalar curvature, to supplement typical techniques for uncertainty quantification, such as Bayesian methods, profile likelihood, asymptotic analysis and bootstrapping. These techniques from information geometry provide data-independent insights into uncertainty and identifiability, and can be used to inform data collection decisions. All code used in this work to implement the inference and information geometry techniques is available on GitHub.

Modeling the popularity of twitter hashtags with master equations.

In this work we introduce a simple mathematical model, based on master equations, to describe the time evolution of the popularity of hashtags on the Twitter social network. Specifically, we model the total number of times a certain hashtag appears on user’s timelines as a function of time. Our model considers two kinds of components: those that are internal to the network (degree distribution) as well as external factors, such as the external popularity of the hashtag. From the master equation, we are able to obtain explicit solutions for the mean and variance and construct confidence regions. We propose a gamma kernel function to model the hashtag popularity, which is quite simple and yields reasonable results. We validate the plausibility of the model by contrasting it with actual Twitter data obtained through the public API. Our findings confirm that relatively simple semi-deterministic models are able to capture the essentials of this very complex phenomenon for a wide variety of cases. The model we present distinguishes from other existing models in its focus on the time evolution of the total number of times a particular hashtag has been seen by Twitter users and the consideration of both internal and external components.

On the computation of confidence regions and error ellipses: a critical appraisal

Customary confidence regions do not truly reflect in the majority of our geodetic applications the confidence one can have in one’s produced estimators. As it is common practice in our daily data analyses to combine methods of parameter estimation and hypothesis testing before the final estimator is produced, it is their combined uncertainty that has to be taken into account when constructing confidence regions. Ignoring the impact of testing on estimation will produce faulty confidence regions and therefore provide an incorrect description of estimator’s quality. In this contribution, we address the interplay between estimation and testing and show how their combined non-normal distribution can be used to construct truthful confidence regions. In doing so, our focus is on the designing phase prior to when the actual measurements are collected, where it is assumed that the working (null) hypothesis is true. We discuss two different approaches for constructing confidence regions: Approach I in which the region’s shape is user-fixed and only its size is determined by the distribution, and Approach II in which both the size and shape are simultaneously determined by the estimator’s non-normal distribution. We also prove and demonstrate that the estimation-only confidence regions have a poor coverage in the sense that they provide an optimistic picture. Next to the provided theory, we provide computational procedures, for both Approach I and Approach II, on how to compute confidence regions and confidence levels that truthfully reflect the combined uncertainty of estimation and testing.

Nonparametric confidence regions via the analytic wild bootstrap

AbstractThe wild bootstrap is a nonparametric tool that can be used to estimate a sampling distribution in the presence of heteroscedastic errors. In particular, the wild bootstrap enables us to compute confidence regions for regression parameters under non‐i.i.d. models. While the wild bootstrap may perform well in these settings, its obvious drawback is a lack of computational efficiency. The wild bootstrap requires a large number of bootstrap replications, making the use of this tool impractical when dealing with big data. We introduce the analytic wild bootstrap (ANWB), which provides a nonparametric alternative way of constructing confidence regions for regression parameters. The ANWB is superior to the wild bootstrap from a computational standpoint while exhibiting similar finite‐sample performance. We report simulation results for both least squares and ridge regression. Additionally, we test the ANWB on a real dataset and compare its performance with that of other standard approaches.

Empirical likelihood for spatial dynamic panel data models with spatial lags and spatial errors

We study spatial dynamic panel data models with both spatial lags and spatial errors. The empirical likelihood (EL) ratio statistics are constructed for the parameters of the models. It is shown that the limiting distributions of the EL ratio statistics are chi-square distributions, which are used to construct confidence regions for the parameters of the models. A simulation study is conducted to compare the performances of the EL based and the normal approximation (NA) based confidence regions. Simulation results show that the EL can be computationally easier than the NA method to implement in practice.

Confidence Regions for Parameters in Stationary Time Series Models With Gaussian Noise

This article develops two new empirical likelihood methods for long-memory time series models based on adjusted empirical likelihood and mean empirical likelihood. By application of Whittle likelihood, one obtains a score function that can be viewed as the estimating equation of the parameters of the long-memory time series model. An empirical likelihood ratio is obtained which is shown to be asymptotically chi-square distributed. It can be used to construct confidence regions. By adding pseudo samples, we simultaneously eliminate the non-definition of the original empirical likelihood and enhance the coverage probability. Finite sample properties of the empirical likelihood confidence regions are explored through Monte Carlo simulation, and some real data applications are carried out.

High-Accuracy Confidence Regions for Distribution Parameters

With the help of today’s computers, it is always relatively easy to find maximum-likelihood estimators of one or more parameters of any specific statistical distribution, and use these to construct the corresponding approximate confidence interval/region, facilitated by the well-known asymptotic properties of the likelihood function. The purpose of this article is to make this approximation substantially more accurate by extending the Taylor expansion of the corresponding probability density function to include quadratic and cubic terms in several centralized sample means, and thus finding the corresponding -proportional correction to the original algorithm. We then demonstrate the new procedure’s usage, both for constructing confidence regions and for testing hypotheses, emphasizing that incorporating this correction carries minimal computational and programming cost. In our final chapter, we present two examples to indicate how significantly the new approximation improves the procedure’s accuracy.

A WILD BOOTSTRAP FOR DEPENDENT DATA

This paper introduces a novel wild bootstrap for dependent data (WBDD) as a means of calculating standard errors of estimators and constructing confidence regions for parameters based on dependent heterogeneous data. The consistency of the bootstrap variance estimator for smooth function of the sample mean is shown to be robust against heteroskedasticity and dependence of unknown form. The first-order asymptotic validity of the WBDD in distribution approximation is established when data are assumed to satisfy a near epoch dependent condition and under the framework of the smooth function model. The WBDD offers a viable alternative to the existing non parametric bootstrap methods for dependent data. It preserves the second-order correctness property of blockwise bootstrap (provided we choose the external random variables appropriately), for stationary time series and smooth functions of the mean. This desirable property of any bootstrap method is not known for extant wild-based bootstrap methods for dependent data. Simulation studies illustrate the finite-sample performance of the WBDD.

Directions Old and New: Palaeomagnetism and Fisher (1953) Meet Modern Statistics

SummaryMost modern articles in the palaeomagnetism literature are based on statistics developed by Fisher's 1953 paper ‘Dispersion on a sphere’, which assumes independent and identically distributed (iid) spherical data. However, palaeomagnetic sample designs are usually hierarchical, where specimens are collected within sites and the data are then combined across sites to calculate an overall mean direction for a geological formation. The specimens within sites are typically more similar than specimens between different sites, and so the iid assumptions fail. This article has three principal goals. The first is to review, contrast and compare both the statistics and geophysics literature on the topic of analysis methods for clustered data on spheres. The second is to present a new hierarchical parametric model, which avoids the unrealistic assumption of rotational symmetry in Fisher's 1953 paper ‘Dispersion on a sphere’ and may be broadly useful in the analysis of many palaeomagnetic datasets. To help develop the model, we use publicly available data as a case study collected from the Golan Heights volcanic plateau. The third goal is to explore different methods for constructing confidence regions for the overall mean direction based on clustered data. Two bootstrap confidence regions that we propose perform well and will be especially useful to geophysics practitioners.

Empirical likelihood for spatial dynamic panel data models.

Spatial dynamic panel data (SDPD) models have received great attention in economics in recent 10 years. Existing approaches for the estimation and test of SDPD models are quasi-maximum likelihood (QML) approach and generalized method of moments (GMM). In this article, we introduce the empirical likelihood (EL) method to the statistical inference for SDPD models. The EL ratio statistics are constructed for the parameters of spatial dynamic panel data models. It is shown that the limiting distributions of the empirical likelihood ratio statistics are chi-squared distributions, which are used to construct confidence regions for the parameters of the models. Simulation results show that the EL based confidence regions outperform the normal approximation based confidence regions.