Infinite-dimensional Nuisance Parameter Research Articles

Semiparametric Bayesian Inference for Local Extrema of Functions in the Presence of Noise

There is a wide range of applications where the local extrema of a function are the key quantity of interest. However, there is surprisingly little work on methods to infer local extrema with uncertainty quantification in the presence of noise. By viewing the function as an infinite-dimensional nuisance parameter, a semiparametric formulation of this problem poses daunting challenges, both methodologically and theoretically, as (i) the number of local extrema may be unknown, and (ii) the induced shape constraints associated with local extrema are highly irregular. In this article, we build upon a derivative-constrained Gaussian process prior recently proposed by Yu et al. to derive what we call an encompassing approach that indexes possibly multiple local extrema by a single parameter. We provide closed-form characterization of the posterior distribution and study its large sample behavior under this unconventional encompassing regime. We show that the posterior measure converges to a mixture of Gaussians with the number of components matching the underlying truth, leading to posterior exploration that accounts for multi-modality. Point and interval estimates of local extrema with frequentist properties are also provided. The encompassing approach leads to a remarkably simple, fast semiparametric approach for inference on local extrema. We illustrate the method through simulations and a real data application to event-related potential analysis. Supplementary materials for this article are available online.

LARGE SAMPLE JUSTIFICATIONS FOR THE BAYESIAN EMPIRICAL LIKELIHOOD

This study investigates the asymptotic properties of the Bayesian empirical likelihood (BEL), which uses the empirical likelihood as an alternative to a parametric likelihood for Bayesian inference. We establish two asymptotic equivalence results based on the Bernstein–von Mises (BvM) theorem by introducing a new formulation of the moment restriction model. First, the limiting posterior distribution of the BEL is the same as that of a parametric Bayesian method that uses the likelihood of a least favorable model of the moment restriction model. Second, the limiting posterior distribution is also the same as that of a semiparametric Bayesian method that places priors on both a finite-dimensional parameter of interest and an infinite-dimensional nuisance parameter. Because parametric and semiparametric Bayesian methods are legitimate Bayesian procedures, the equivalence results provide a large sample justification for the BEL as a Bayesian inference method. Moreover, the BvM theorem provides a frequentist justification for BEL posterior inference.

Transformation-Invariant Learning of Optimal Individualized Decision Rules with Time-to-Event Outcomes

In many important applications of precision medicine, the outcome of interest is time to an event (e.g., death, relapse of disease) and the primary goal is to identify the optimal individualized decision rule (IDR) to prolong survival time. Existing work in this area have been mostly focused on estimating the optimal IDR to maximize the restricted mean survival time in the population. We propose a new robust framework for estimating an optimal static or dynamic IDR with time-to-event outcomes based on an easy-to-interpret quantile criterion. The new method does not need to specify an outcome regression model and is robust for heavy-tailed distribution. The estimation problem corresponds to a nonregular M-estimation problem with both finite and infinite-dimensional nuisance parameters. Employing advanced empirical process techniques, we establish the statistical theory of the estimated parameter indexing the optimal IDR. Furthermore, we prove a novel result that the proposed approach can consistently estimate the optimal value function under mild conditions even when the optimal IDR is nonunique, which happens in the challenging setting of exceptional laws. We also propose a smoothed resampling procedure for inference. The proposed methods are implemented in the R-package QTOCen. We demonstrate the performance of the proposed new methods via extensive Monte Carlo studies and a real data application. Supplementary materials for this article are available online.

Testing subspace restrictions in the presence of high dimensional nuisance parameters

A framework for hypothesis testing of functional restrictions against general alternatives is proposed. The parameter space is a subset of a reproducing kernel Hilbert space (RKHS). The null hypothesis does not necessarily define a parametric model. The test allows us to deal with possibly infinite dimensional nuisance parameters. The methodology is based on a moment equation similar in spirit to the construction of the efficient score in semiparametric statistics. The feasible version of such moment equation requires to consistently estimate projections in the space of RKHS. A tractable asymptotic theory is established for this problem. Simulation results show that the finite sample performance of the test is consistent with the asymptotic results and that ignoring the effect of nuisance parameters highly distorts the size of the tests.

Joint Estimation of Location and Scatter in Complex Elliptically Symmetric Distributions

The joint estimation of the location vector and the shape matrix of a set of independent and identically Complex Elliptically Symmetric (CES) distributed observations is investigated from both the theoretical and computational viewpoints. This joint estimation problem is framed in the original context of semiparametric models allowing us to handle the (generally unknown) density generator as an infinite-dimensional nuisance parameter. In the first part of the paper, a computationally efficient and memory saving implementation of the robust and semiparmaetric efficient R-estimator for shape matrices is derived. Building upon this result, in the second part, a joint estimator, relying on the Tyler’s M-estimator of location and on the R-estimator of shape matrix, is proposed and its Mean Squared Error (MSE) performance compared with the Semiparametric Cramér-Rao Bound (SCRB).

Semiparametric testing with highly persistent predictors

We address the issue of semiparametric efficiency in the bivariate regression problem with a highly persistent predictor, where the joint distribution of the innovations is regarded an infinite-dimensional nuisance parameter. Using a structural representation of the limit experiment and exploiting invariance relationships therein, we construct invariant point-optimal tests for the regression coefficient of interest. This approach naturally leads to a family of feasible tests based on the component-wise ranks of the innovations that can gain considerable power relative to existing tests under non-Gaussian innovation distributions, while behaving equivalently under Gaussianity. When an i.i.d. assumption on the innovations is appropriate for the data at hand, our tests exploit the efficiency gains possible. Moreover, we show by simulation that our test remains well behaved under some forms of conditional heteroskedasticity.

Large Sample Justifications for the Bayesian Empirical Likelihood

This study investigates the asymptotic properties of the Bayesian empirical likelihood (BEL), which uses the empirical likelihood as an alternative to a parametric likelihood for Bayesian inference. We establish two asymptotic equivalence results based on the Bernstein-von Mises (BvM) theorem by introducing a new formulation of the moment restriction model. First, the limiting posterior distribution of the BEL is the same as that of a parametric Bayesian method that uses the likelihood of a least favorable model of the moment restriction model. Second, the limiting posterior distribution is also the same as that of a semi-parametric Bayesian method that places priors on both a finite-dimensional parameter of interest and an infinite-dimensional nuisance parameter. Because parametric and semi-parametric Bayesian methods are legitimate Bayesian procedures, the equivalence results provide a large sample justification for the BEL as a Bayesian inference method. Moreover, the BvM theorem guarantees the validity of the BEL as a frequentist estimator.

Sensitivity Analysis in Semiparametric Likelihood Models

We provide methods for inference on a finite dimensional parameter of interest, � 2 < d�, in a semiparametric probability model when an infinite dimensional nuisance parameter, g, is present. We depart from the semiparametric literature in that we do not require that the pair (�,g) is point identified and so we construct confidence regions forthat are robust to non-point identification. This allows practitioners to examine the sensitivity of their estimates ofto specification of g in a likelihood setup. To construct these confidence regions for �, we invert a profiled sieve likelihood ratio (LR) statistic. We derive the asymptotic null distribution of this profiled sieve LR, which is nonstandard whenis not point identified (but is � 2 distributed under point identifica- tion). We show that a simple weighted bootstrap procedure consistently estimates this complicated distribution's quantiles. Monte Carlo studies of a semiparametric dynamic binary response panel data model indicate that our weighted bootstrap procedures per- forms adequately in finite samples. We provide three empirical illustrations where we compare our results to the ones obtained using standard (less robust) methods.

Model-free posterior inference on the area under the receiver operating characteristic curve

The area under the receiver operating characteristic curve (AUC) serves as a summary of a binary classifier’s performance. For inference on the AUC, a common modeling assumption is binormality, which restricts the distribution of the score produced by the classifier. However, this assumption introduces an infinite-dimensional nuisance parameter and may be restrictive in certain machine learning settings. To avoid making distributional assumptions, and to avoid the computational challenges of a fully nonparametric analysis, we develop a direct and model-free Gibbs posterior distribution for inference on the AUC. We present the asymptotic Gibbs posterior concentration rate, and a strategy for tuning the learning rate so that the corresponding credible intervals achieve the nominal frequentist coverage probability. Simulation experiments and a real data analysis demonstrate the Gibbs posterior’s strong performance compared to existing Bayesian methods.

Optimal tests for parameter breaking process in conditional quantile models

This paper proposes efficient tests for quantile parameter instability in parametric and semiparametric setups. In each setup, various types of unstable parameter processes are examined such as single structural break, multiple structural breaks, and random parameters, and the optimal test is suggested for each unstable process. In a parametric model, tick-exponential family of distributions is used to construct the likelihood ratio tests. The suggested tests have the best asymptotic weighted average power if the likelihood function is correctly specified and are asymptotically correct-sized even under misspecification. In a semiparametric setup in which the underlying distribution is unknown but is treated as an infinite-dimensional nuisance parameter, we show that semiparametric efficient tests are adaptive if the error term is conditionally iid. Non-adaptive efficient tests are suggested under weaker conditions as well. Monte Carlo simulation shows that the proposed tests have better finite sample powers than the existing tests under various circumstances.

Construction of credible intervals for nonlinear regression models with unknown error distributions

ABSTRACTThere has been continuing interest in Bayesian regressions in which no parametric assumptions are imposed on the error distribution. In this study, we consider semiparametric Bayesian nonlinear regression models. We do not impose a parametric form for the likelihood function. Instead, we treat the true density function of error terms as an infinite-dimensional nuisance parameter and estimate it nonparametrically. Thereafter, we conduct a conventional parametric Bayesian inference using MCMC methods. We derive the asymptotic properties of the resulting estimator and identify conditions for adaptive estimation, under which our two-step Bayes estimator enjoys the same asymptotic efficiency as if we knew the true density. We compare the accuracy and coverage of the adaptive Bayesian point and interval estimators to those of the maximum likelihood estimator empirically, using simulated and real data. In particular, we observe that the Bayesian inference may be superior in numerical stability for small sample sizes.

An Adapted Loss Function for Censored Quantile Regression

ABSTRACTIn this article, we study a novel approach for the estimation of quantiles when facing potential right censoring of the responses. Contrary to the existing literature on the subject, the adopted strategy of this article is to tackle censoring at the very level of the loss function usually employed for the computation of quantiles, the so-called “check” function. For interpretation purposes, a simple comparison with the latter reveals how censoring is accounted for in the newly proposed loss function. Subsequently, when considering the inclusion of covariates for conditional quantile estimation, by defining a new general loss function the proposed methodology opens the gate to numerous parametric, semiparametric, and nonparametric modeling techniques. To illustrate this statement, we consider the well-studied linear regression under the usual assumption of conditional independence between the true response and the censoring variable. For practical minimization of the studied loss function, we also provide a simple algorithmic procedure shown to yield satisfactory results for the proposed estimator with respect to the existing literature in an extensive simulation study. From a more theoretical prospect, consistency and asymptotic normality of the estimator for linear regression are obtained using several recent results on nonsmooth semiparametric estimation equations with an infinite-dimensional nuisance parameter, while numerical examples illustrate the adequateness of a simple bootstrap procedure for inferential purposes. Lastly, an application to a real dataset is used to further illustrate the validity and finite sample performance of the proposed estimator. Supplementary materials for this article are available online.

Semiparametric M-estimation with non-smooth criterion functions

We are interested in the estimation of a parameter θ that maximizes a certain criterion function depending on an unknown, possibly infinite dimensional nuisance parameter h. A common estimation procedure consists in maximizing the corresponding empirical criterion, in which the nuisance parameter is replaced by a nonparametric estimator. In the literature, this research topic, commonly referred to as semiparametric M-estimation, has received a lot of attention in the case where the criterion function M satisfies certain smoothness properties. In certain applications, these smoothness conditions are however not satisfied. The aim of this paper is therefore to extend the existing theory on semiparametric M-estimation problems, in order to cover non-smooth M-estimation problems as well. In particular, we develop `high-level' conditions under which the proposed M-estimator is consistent and has an asymptotic limit. We also check these conditions in detail for a specific example of a semiparametric M-estimation problem, which comes from the area of classification with missing data, and which cannot be dealt with using the existing results in the literature.

Semiparametric Optimal Testing With Highly Persistent Predictors

We address the issue of semiparametric efficiency in the bivariate regression problem with a highly persistent predictor, where the joint distribution of the innovations is regarded an infinite-dimensional nuisance parameter. Using a structural representation of the limit experiment and exploiting invariance relationships therein, we construct invariant point-optimal tests for the regression coefficient of interest. This approach naturally leads to a family of feasible tests based on the component-wise ranks of the innovations that can gain considerable power relative to existing tests under non-Gaussian innovation distributions, while behaving equivalently under Gaussianity. When an i.i.d. assumption on the innovations is appropriate for the data at hand, our tests exploit the efficiency gains possible. Moreover, we show by simulation that our test remains well behaved under some forms of conditional heteroskedasticity.

Outcome-dependent sampling with interval-censored failure time data.

Epidemiologic studies and disease prevention trials often seek to relate an exposure variable to a failure time that suffers from interval-censoring. When the failure rate is low and the time intervals are wide, a large cohort is often required so as to yield reliable precision on the exposure-failure-time relationship. However, large cohort studies with simple random sampling could be prohibitive for investigators with a limited budget, especially when the exposure variables are expensive to obtain. Alternative cost-effective sampling designs and inference procedures are therefore desirable. We propose an outcome-dependent sampling (ODS) design with interval-censored failure time data, where we enrich the observed sample by selectively including certain more informative failure subjects. We develop a novel sieve semiparametric maximum empirical likelihood approach for fitting the proportional hazards model to data from the proposed interval-censoring ODS design. This approach employs the empirical likelihood and sieve methods to deal with the infinite-dimensional nuisance parameters, which greatly reduces the dimensionality of the estimation problem and eases the computation difficulty. The consistency and asymptotic normality of the resulting regression parameter estimator are established. The results from our extensive simulation study show that the proposed design and method works well for practical situations and is more efficient than the alternative designs and competing approaches. An example from the Atherosclerosis Risk in Communities (ARIC) study is provided for illustration.

Minimum distance from independence estimation of nonseparable instrumental variables models

I develop a semiparametric minimum distance from independence estimator for a nonseparable instrumental variables model. An independence condition identifies the model for many types of discrete and continuous instruments. The estimator is taken as the parameter value that most closely satisfies this independence condition. Implementing the estimator requires a quantile regression of the endogenous variables on the instrument, so the procedure is two-step, with a finite or infinite-dimensional nuisance parameter in the first step. I prove consistency and establish asymptotic normality for a parametric, but flexibly nonlinear outcome equation. The consistency of the nonparametric bootstrap is also shown. I illustrate the use of the estimator by estimating the returns to schooling using data from the 1979 National Longitudinal Survey.

Empirical likelihood confidence regions in the single-index model with growing dimensions

ABSTRACTThis paper investigates statistical inference for the single-index model when the number of predictors grows with sample size. Empirical likelihood method for constructing confidence region for the index vector, which does not require a multivariate non parametric smoothing, is employed. However, the classical empirical likelihood ratio for this model does not remain valid because plug-in estimation of an infinite-dimensional nuisance parameter causes a non negligible bias and the diverging number of parameters/predictors makes the limit not chi-squared any more. To solve these problems, we define an empirical likelihood ratio based on newly proposed weighted estimating equations and show that it is asymptotically normal. Also we find that different weights used in the weighted residuals require, for asymptotic normality, different diverging rate of the number of predictors. However, the rate n1/3, which is a possible fastest rate when there are no any other conditions assumed in the setting under study, is still attainable. A simulation study is carried out to assess the performance of our method.

Adaptive Bayesian Inference in Nonlinear Regression Models

There has been continuing interest in Bayesian regressions without imposing any parametric assumption on the error distribution, but the asymptotic efficiency of such procedures has not been fully understood yet. In this article, we consider semiparametric Bayesian nonlinear regression models. We do not impose a parametric form for the likelihood function; rather, we treat the true density function of error terms as an infinite dimensional nuisance parameter and estimate it nonparametrically. Thereafter, we conduct conventional parametric Bayesian inference using MCMC methods. We derive the asymptotic properties of the resulting estimator and identify conditions of adaptive estimation, under which our two-step Bayes estimator enjoys the same asymptotic normality as if we knew the true density. We compare accuracy and coverage of the adaptive Bayesian estimator with the maximum likelihood estimator in empirical studies on simulated and real data. In particular, we observe that the Bayesian inference may be superior in numerical stability for small sample sizes.

On pseudolikelihood inference for semiparametric models with boundary problems.

SUMMARYConsider a semiparametric model indexed by a Euclidean parameter of interest and an infinite-dimensional nuisance parameter. In many applications, pseudolikelihood provides a convenient way to infer the parameter of interest, where the nuisance parameter is replaced by a consistent estimator. The purpose of this paper is to establish the asymptotic behaviour of the pseudolikelihood ratio statistic under semiparametric models. In particular, we consider testing the hypothesis that the parameter of interest lies on the boundary of its parameter space. Under regularity conditions, we establish the equivalence between the asymptotic distributions of the pseudolikelihood ratio statistic and a likelihood ratio statistic for a normal mean problem with a misspecified covariance matrix. This result holds when the nuisance parameter is estimated at a rate slower than the usual rate in parametric models. We study three examples in which the asymptotic distributions are shown to be mixtures of chi-squared variables. We conduct simulation studies to examine the finite-sample performance of the pseudolikelihood ratio test.

Data-Adaptive Bias-Reduced Doubly Robust Estimation.

Doubly robust estimators have now been proposed for a variety of target parameters in the causal inference and missing data literature. These consistently estimate the parameter of interest under a semiparametric model when one of two nuisance working models is correctly specified, regardless of which. The recently proposed bias-reduced doubly robust estimation procedure aims to partially retain this robustness in more realistic settings where both working models are misspecified. These so-called bias-reduced doubly robust estimators make use of special (finite-dimensional) nuisance parameter estimators that are designed to locally minimize the squared asymptotic bias of the doubly robust estimator in certain directions of these finite-dimensional nuisance parameters under misspecification of both parametric working models. In this article, we extend this idea to incorporate the use of data-adaptive estimators (infinite-dimensional nuisance parameters), by exploiting the bias reduction estimation principle in the direction of only one nuisance parameter. We additionally provide an asymptotic linearity theorem which gives the influence function of the proposed doubly robust estimator under correct specification of a parametric nuisance working model for the missingness mechanism/propensity score but a possibly misspecified (finite- or infinite-dimensional) outcome working model. Simulation studies confirm the desirable finite-sample performance of the proposed estimators relative to a variety of other doubly robust estimators.