Data-dependent Choice Research Articles

FsdaSAS: A Package for Robust Regression for Very Large Datasets Including the Batch Forward Search

The forward search (FS) is a general method of robust data fitting that moves smoothly from very robust to maximum likelihood estimation. The regression procedures are included in the MATLAB toolbox FSDA. The work on a SAS version of the FS originates from the need for the analysis of large datasets expressed by law enforcement services operating in the European Union that use our SAS software for detecting data anomalies that may point to fraudulent customs returns. Specific to our SAS implementation, the fsdaSAS package, we describe the approximation used to provide fast analyses of large datasets using an FS which progresses through the inclusion of batches of observations, rather than progressing one observation at a time. We do, however, test for outliers one observation at a time. We demonstrate that our SAS implementation becomes appreciably faster than the MATLAB version as the sample size increases and is also able to analyse larger datasets. The series of fits provided by the FS leads to the adaptive data-dependent choice of maximally efficient robust estimates. This also allows the monitoring of residuals and parameter estimates for fits of differing robustness levels. We mention that our fsdaSAS also applies the idea of monitoring to several robust estimators for regression for a range of values of breakdown point or nominal efficiency, leading to adaptive values for these parameters. We have also provided a variety of plots linked through brushing. Further programmed analyses include the robust transformations of the response in regression. Our package also provides the SAS community with methods of monitoring robust estimators for multivariate data, including multivariate data transformations.

Inference after estimation of breaks

In an important class of econometric problems, researchers select a target parameter by maximizing the Euclidean norm of a data-dependent vector. Examples that can be cast into this frame include threshold regression models with estimated thresholds and structural break models with estimated break dates. Estimation and inference procedures that ignore the randomness of the target parameter can be severely biased and misleading when this randomness is non-negligible. This paper studies conditional and unconditional inference in such settings, accounting for the data-dependent choice of target parameters. We detail the construction of quantile-unbiased estimators and confidence sets with correct coverage, and prove their asymptotic validity under data generating process such that the target parameter remains random in the limit. We also provide a novel sample splitting approach that improves on conventional split-sample inference.

Breaking the Bandwidth Barrier: Geometrical Adaptive Entropy Estimation

Estimators of information theoretic measures, such as entropy and mutual information, are a basic workhorse for many downstream applications in modern data science. State-of-the-art approaches have been either geometric [nearest neighbor (NN)-based] or kernel-based (with a globally chosen bandwidth). In this paper, we combine both these approaches to design new estimators of entropy and mutual information that outperform the state-of-the-art methods. Our estimator uses local bandwidth choices of $k$ -NN distances with a finite $k$ , independent of the sample size. Such a local and data dependent choice ameliorates boundary bias and improves performance in practice, but the bandwidth is vanishing at a fast rate, leading to a non-vanishing bias. We show that the asymptotic bias of the proposed estimator is universal ; it is independent of the underlying distribution. Hence, it can be precomputed and subtracted from the estimate. As a byproduct, we obtain a unified way of obtaining both the kernel and NN estimators. The corresponding theoretical contribution relating the asymptotic geometry of nearest neighbors to order statistics is of independent mathematical interest.

An ‘apples to apples’ comparison of various tests for exponentiality

The exponential distribution is a popular model both in practice and in theoretical work. As a result, a multitude of tests based on varied characterisations have been developed for testing the hypothesis that observed data are realised from this distribution. Many of the recently developed tests contain a tuning parameter, usually appearing in a weight function. In this paper we compare the powers of 20 tests for exponentiality—some containing a tuning parameter and some that do not. To ensure a fair ‘apples to apples’ comparison between each of the tests, we employ a data-dependent choice of the tuning parameter for those tests that contain these parameters. The comparisons are conducted for various samples sizes and for a large number of alternative distributions. The results of the simulation study show that the test with the best overall power performance is the Baringhaus and Henze test, followed closely by the test by Henze and Meintanis; both tests contain a tuning parameter. The score test by Cox and Oakes performs the best among those tests that do not include a tuning parameter.

Smooth-Threshold Multivariate Genetic Prediction with Unbiased Model Selection.

We develop a new genetic prediction method, smooth-threshold multivariate genetic prediction, using single nucleotide polymorphisms (SNPs) data in genome-wide association studies (GWASs). Our method consists of two stages. At the first stage, unlike the usual discontinuous SNP screening as used in the gene score method, our method continuously screens SNPs based on the output from standard univariate analysis for marginal association of each SNP. At the second stage, the predictive model is built by a generalized ridge regression simultaneously using the screened SNPs with SNP weight determined by the strength of marginal association. Continuous SNP screening by the smooth thresholding not only makes prediction stable but also leads to a closed form expression of generalized degrees of freedom (GDF). The GDF leads to the Stein's unbiased risk estimation (SURE), which enables data-dependent choice of optimal SNP screening cutoff without using cross-validation. Our method is very rapid because computationally expensive genome-wide scan is required only once in contrast to the penalized regression methods including lasso and elastic net. Simulation studies that mimic real GWAS data with quantitative and binary traits demonstrate that the proposed method outperforms the gene score method and genomic best linear unbiased prediction (GBLUP), and also shows comparable or sometimes improved performance with the lasso and elastic net being known to have good predictive ability but with heavy computational cost. Application to whole-genome sequencing (WGS) data from the Alzheimer's Disease Neuroimaging Initiative (ADNI) exhibits that the proposed method shows higher predictive power than the gene score and GBLUP methods.

Sample size reassessment for a two-stage design controlling the false discovery rate.

Sample size calculations for gene expression microarray and NGS-RNA-Seq experiments are challenging because the overall power depends on unknown quantities as the proportion of true null hypotheses and the distribution of the effect sizes under the alternative. We propose a two-stage design with an adaptive interim analysis where these quantities are estimated from the interim data. The second stage sample size is chosen based on these estimates to achieve a specific overall power. The proposed procedure controls the power in all considered scenarios except for very low first stage sample sizes. The false discovery rate (FDR) is controlled despite of the data dependent choice of sample size. The two-stage design can be a useful tool to determine the sample size of high-dimensional studies if in the planning phase there is high uncertainty regarding the expected effect sizes and variability.

On a data-dependent choice of the tuning parameter appearing in certain goodness-of-fit tests

We propose a data-dependent method for choosing the tuning parameter appearing in many recently developed goodness-of-fit test statistics. The new method, based on the bootstrap, is applicable to a class of distributions for which the null distribution of the test statistic is independent of unknown parameters. No data-dependent choice for this parameter exists in the literature; typically, a fixed value for the parameter is chosen which can perform well for some alternatives, but poorly for others. The performance of the new method is investigated by means of a Monte Carlo study, employing three tests for exponentiality. It is found that the Monte Carlo power of these tests, using the data-dependent choice, compares favourably to the maximum achievable power for the tests calculated over a grid of values of the tuning parameter.

Convergence and Consistency of Regularized Boosting With Weakly Dependent Observations

This paper studies the statistical convergence and consistency of regularized boosting methods, where the samples need not be independent and identically distributed but can come from stationary weakly dependent sequences. Consistency is proven for the composite classifiers that result from a regularization achieved by restricting the 1-norm of the base classifiers' weights. The less restrictive nature of sampling considered here is manifested in the consistency result through a generalized condition on the growth of the regularization parameter. The weaker the sample dependence, the faster the regularization parameter is allowed to grow with increasing sample size. A consistency result is also provided for data-dependent choices of the regularization parameter.

Two new data-dependent choices of m when applying the m-out-of-n bootstrap to hypothesis testing

The traditional non-parametric bootstrap (referred to as the n-out-of-n bootstrap) is a widely applicable and powerful tool for statistical inference, but in important situations it can fail. It is well known that by using a bootstrap sample of size m, different from n, the resulting m-out-of-n bootstrap provides a method for rectifying the traditional bootstrap inconsistency. Moreover, recent studies have shown that interesting cases exist where it is better to use the m-out-of-n bootstrap in spite of the fact that the n-out-of-n bootstrap works. In this paper, we discuss another case by considering its application to hypothesis testing. Two new data-based choices of m are proposed in this set-up. The results of simulation studies are presented to provide empirical comparisons between the performance of the traditional bootstrap and the m-out-of-n bootstrap, based on the two data-dependent choices of m, as well as on an existing method in the literature for choosing m. These results show that the m-out-of-n bootstrap, based on our choice of m, generally outperforms the traditional bootstrap procedure as well as the procedure based on the choice of m proposed in the literature.

Outlier Detection Using Nonconvex Penalized Regression

This article studies the outlier detection problem from the standpoint of penalized regression. In the regression model, we add one mean shift parameter for each of the n data points. We then apply a regularization favoring a sparse vector of mean shift parameters. The usual L1 penalty yields a convex criterion, but fails to deliver a robust estimator. The L1 penalty corresponds to soft thresholding. We introduce a thresholding (denoted by Θ) based iterative procedure for outlier detection (Θ–IPOD). A version based on hard thresholding correctly identifies outliers on some hard test problems. We describe the connection between Θ–IPOD and M-estimators. Our proposed method has one tuning parameter with which to both identify outliers and estimate regression coefficients. A data-dependent choice can be made based on the Bayes information criterion. The tuned Θ–IPOD shows outstanding performance in identifying outliers in various situations compared with other existing approaches. In addition, Θ–IPOD is much faster than iteratively reweighted least squares for large data, because each iteration costs at most O(np) (and sometimes much less), avoiding an O(np2) least squares estimate. This methodology can be extended to high-dimensional modeling with p ≫ n if both the coefficient vector and the outlier pattern are sparse.

Robust Shrinkage Estimation of High-Dimensional Covariance Matrices

We address high dimensional covariance estimation for elliptical distributed samples, which are also known as spherically invariant random vectors (SIRV) or compound-Gaussian processes. Specifically we consider shrinkage methods that are suitable for high dimensional problems with a small number of samples (large $p$ small $n$). We start from a classical robust covariance estimator [Tyler(1987)], which is distribution-free within the family of elliptical distribution but inapplicable when $n<p$. Using a shrinkage coefficient, we regularize Tyler's fixed point iterations. We prove that, for all $n$ and $p$, the proposed fixed point iterations converge to a unique limit regardless of the initial condition. Next, we propose a simple, closed-form and data dependent choice for the shrinkage coefficient, which is based on a minimum mean squared error framework. Simulations demonstrate that the proposed method achieves low estimation error and is robust to heavy-tailed samples. Finally, as a real world application we demonstrate the performance of the proposed technique in the context of activity/intrusion detection using a wireless sensor network.

Continuum or community: a priori assumption or data-dependent choice?

This paper examines how we might test the continuum theory against the community unit theory. Adherence to one or other of these models without testing is simply an assignment of an extreme prior probability to the preferred option. The question can be rephrased to ask whether, for a set of observations, a single model is adequate or whether a mixture of models would be preferable. To judge between them involves first defining the nature of the model(s) to be fitted in each case and then comparing the complexity and quality of fit. Occam’s razor suggests that we should seek the simplest model with adequate fit, with parameters estimated with optimal precision. The simplest comparison of the two theories thus requires only the estimation of the number of clusters for the chosen model(s) of within-cluster variation. If a single cluster is of adequate quality then the continuum model is appropriate, while if several are needed then the community model is preferable for that particular dataset. To establish universal applicability of either model involves investigation of many datasets.

The use of quad–quad resistivity in helicopter electromagnetic mapping

The apparent resistivity from a helicopter-borne frequency-domain electromagnetic (EM) system is typically obtained from the in-phase and quadrature responses arising from the flow of conduction currents in the earth. The most commonly used resistivity algorithms, derived from half-space models and using single-frequency data, do not account for magnetic polarization and consequently do not yield a reliable value for apparent resistivity in highly magnetic areas. This is because magnetic polarization modifies the EM response, causing the computed resistivity to be erroneously high. The impact of magnetic permeability on the EM response is much greater for the in-phase component than for the quadrature component. If magnetic polarization is to be ignored, the calculation of the apparent resistivity using the quadrature component at two frequencies (the quad–quad algorithm) is less subject to error from magnetic polarization than if the in-phase and quadrature responses at a single frequency are used (the in-phase–quad algorithm). The quad–quad algorithm, however, can display undesirable behavior for large induction numbers, i.e., when conductivities and frequencies are large. Determining which algorithm is optimum is a data-dependent choice, which, of course, is area dependent. We have studied the behavior of the quad–quad (apparent) resistivity and its comparison to in-phase–quad resistivity to determine the conditions under which the use of quad–quad resistivity is appropriate. For a two-layer earth, the behavior of the quad–quad resistivity depends mainly upon the ratio of the lower frequency fL to the upper-layer resistivity ρ1. If this ratio is low, the quad–quad resistivity will behave well. In areas yielding a high value of the ratio fL/ρ1, the quad–quad resistivity may lie outside of the range of the true resistivities of the earth and therefore provide misleading information. Our studies therefore suggest that the quad–quad resistivity algorithm should be avoided in areas where the ratio is large, i.e., when using high frequencies in conductive areas. The term large is relative. For a two-layer case, for example, the use of quad–quad resistivity is only recommended for magnetic areas where fL/ρ1 &lt; 500 Hz/ohm-m, when conductive cover exists, and where fL/ρ1 &lt; 50 Hz/ohm-m when a conductive basement underlies resistive cover. In spite of these limitations, quad–quad resistivity is often preferable to in-phase–quad resistivity in highly magnetic areas.

Adaptive Semiparametric Estimation of the Memory Parameter

In Giraitis, Robinson, and Samarov (1997), we have shown that the optimal rate for memory parameter estimators in semiparametric long memory models with degree of “local smoothness” β is n−r(β), r(β)=β/(2β+1), and that a log-periodogram regression estimator (a modified Geweke and Porter-Hudak (1983) estimator) with maximum frequency m=m(β)≍n2r(β) is rate optimal. The question which we address in this paper is what is the best obtainable rate when β is unknown, so that estimators cannot depend on β. We obtain a lower bound for the asymptotic quadratic risk of any such adaptive estimator, which turns out to be larger than the optimal nonadaptive rate n−r(β) by a logarithmic factor. We then consider a modified log-periodogram regression estimator based on tapered data and with a data-dependent maximum frequency m=m(β), which depends on an adaptively chosen estimator β of β, and show, using methods proposed by Lepskii (1990) in another context, that this estimator attains the lower bound up to a logarithmic factor. On one hand, this means that this estimator has nearly optimal rate among all adaptive (free from β) estimators, and, on the other hand, it shows near optimality of our data-dependent choice of the rate of the maximum frequency for the modified log-periodogram regression estimator. The proofs contain results which are also of independent interest: one result shows that data tapering gives a significant improvement in asymptotic properties of covariances of discrete Fourier transforms of long memory time series, while another gives an exponential inequality for the modified log-periodogram regression estimator.

Simultaneous bootstrap confidence bands in nonparametric regression

In the present paper we construct asymptotic confidence bands in non-parametric regression. Our assumptions cover unequal variances of the observations and nonuni-form, possibly considerably clustered design. The confidence band is based on an undersmoothed local polynomial estimator. An appropriate quantile is obtained via the wild bootstrap. We derive certain rates (in the sample size n) for the error in coverage probability, which improves on existing results for methods that rely on the asymptotic distribution of the maximum of some Gaussian process. We propose a practicable rule for a data-dependent choice of the band-width. A small simulation study illustrates the possible gains by our approach over alternative frequently used methods.

Data driven smooth tests for composite hypotheses comparison of powers

Rao's score statistic is a standard tool for constructing statistical tests.If departures from the null model are described by some k-dimensional exponential family the resulting score test is called also smooth test or Neyman's smooth test with k components. An important practical question in applying a smooth test in the goodness-of-fit problem is how large k should be taken. Since a wrong choice may give a considerable loss of power,it is important to make a careful selection.Renewed research in this area shows that the simple question has no simple deterministic answer. Therefore,edwina introduced,for testing a simple goodness-of-fit hypothesis,a data driven version of Neyman's smooth test. First,Schwarz's rule is applied to find a suitable dimension and then the smooth test statistic in the “right” dimension finishes the job. Simulation results and some theoretical considerations show that this data driven version of smooth tests performs well for a wide range of alternatives,and is competitive with other recently introduced (data driven) procedures.This data-dependent choice of the number of components is extended in this paper to testing the goodness-of-fit problem with composite null hypothesis,being of more practical interest. A k-dimensional exponential family for modelling departures from the null hypothesis is given and the related Rao's score test is described. A suitable version of Schwarz's rule is proposed and some simplifications of it are discussed. To check validity of the proposed construction,the method is applied to testing exponentiality and normality.In the extensive simulation study,presented in this paper,it turns out that the data driven version of smooth tests compares well for a wide range of alternatives with other,more specialized,commonly used tests.

Data-dependent bandwidth selection for emission computed tomography reconstruction

At present, the choice of bandwidth in emission computed tomography (ECT) reconstruction is done by subjective means. The authors develop an automated objective selection technique for linear reconstruction algorithms such as filtered backprojection. The approach is based on the method of unbiased risk estimation. A set of 2-D validation studies using computer simulated and physical phantom data from the Hoffman et al. (1990) brain phantom are carried out. These 2-D studies incorporate measured corrections for object attenuation and lack of uniformity in detector sensitivity. It is found that the unbiased risk approach works very well. Over a range of count rates and brain slice source distributions, the root mean square (RMS) error of the fully automated reconstruction, with the data-dependent choice of bandwidth, is around 5% greater than the RMS error for the reconstruction with an ideal choice of the bandwidth.