Plug-in Rule Research Articles

Independent rule in classification of multivariate binary data

We consider the performance of the independent rule in classification of multivariate binary data. In this article, broad studies are presented including the performance of the independent rule when the number of variables, d , is fixed or increased with the sample size, n . The latter situation includes the case of d = O ( n τ ) for τ > 0 which cover “the small sample and the large dimension”, namely d ≫ n when τ > 1 . Park and Ghosh [J. Park, J.K. Ghosh, Persistence of plug-in rule in classification of high dimensional binary data, Journal of Statistical Planning and Inference 137 (2007) 3687–3707] studied the independent rule in terms of the consistency of misclassification error rate which is called persistence under growing numbers of dimensions, but they did not investigate the convergence rate. We present asymptotic results in view of the convergence rate under some structured parameter space and highlight that variable selection is necessary to improve the performance of the independent rule. We also extend the applications of the independent rule to the case of correlated binary data such as the Bahadur representation and the logit model. It is emphasized that variable selection is also needed in correlated binary data for the improvement of the performance of the independent rule.

ANALYSIS OF CLASSIFICATION WITH A REJECT OPTION

In many classification problems, objects should be rejected when the confidence in their classification is too low. In this paper, we consider a new classification algorithm with a reject option. Based on the majority vote strategy and plug-in rules, we provide error analysis for this algorithm in ideal and realistic settings, respectively. In addition, some discussions of semi-supervised classification are given to demonstrate our theoretical analysis.

Automatic bandwidth selection for circular density estimation

Given angular data θ 1 , … , θ n ∈ [ 0 , 2 π ) a common objective is to estimate the density. In case that a kernel estimator is used, bandwidth selection is crucial to the performance. A “plug-in rule” for the bandwidth, which is based on the concentration of a reference density, namely, the von Mises distribution is obtained. It is seen that this is equivalent to the usual Euclidean plug-in rule in the case where the concentration becomes large. In case that the concentration parameter is unknown, alternative methods are explored which are intended to be robust to departures from the reference density. Simulations indicate that “wrapped estimators” can perform well in this context. The methods are applied to a real bivariate dataset concerning protein structure.

A simple and effective bandwidth selector for local polynomial quasi-likelihood regression

Local quasi-likelihood methods are powerful nonparametric techniques that can be applied to a variety of regression problems where the conventional least squares approach is not appropriate. They are particularly useful for analyzing regression data with binary and count responses. In this paper, we propose a new bandwidth selector for local quasi-likelihood regression estimation. It eschews conventional cross-validation that requires fitting the data repeatedly with one or some of the data leaved out. Our proposal needs only a single fit of the whole regression function, and does not call for selection of secondary tuning parameters as in plug-in rules. The method is based on a uniform stochastic expansion for the estimated quasi-likelihood, which we derive in this paper. We investigate the finite sample properties of the proposed bandwidth selector through a Monte Carlo simulation.

Persistence of plug-in rule in classification of high dimensional multivariate binary data

In this paper, we consider the classification problem when the predictors are multivariate binary random variables. Variables are modeled as independent, but not necessarily identical, Bernoulli. A triangular array for parameters, ( p 11 ( n ) , … , p 1 d ( n ) , p 21 ( n ) , … , p 2 d ( n ) ) , is assumed to allow parameters to change and the number of the variables, d, to increase for adopting more flexible models as the sample size, n, increases. Our results are obtained under moderate assumptions on the triangular array of the probability vectors. We use maximum likelihood estimators for the parameters and plug them into the Bayes classifier. This is a plug-in classifier, a sort of objective Bayes rule. It is shown in Wilbur et al. [2002. Variable selection in high-dimensional multivariate binary data with application to the analysis of microbial DNA fingerprints. Biometrics 58, 378–386] via simulations that the plug-in rule classifies quite well even when the assumption of independence is violated. The main interest in this paper is in the complex case of d / n ν → c for some ν > 0 and c > 0 for which very little is known. Using linearity of the plug-in rule, we show its persistence, a generalization of the notion of consistency, when the variance of the plug-in rule or a quantity measuring signal to noise ratio is divergent; otherwise we show there exists an example of non-persistence of the plug-in rule. In case of non-persistence, we introduce the notion of sparsity and overcome non-persistence by selecting a subset of the variables. This shows why a variable selection procedure may be effective especially for contemporary practical problems with high dimensional data [Wilbur et al., 2002. Variable selection in high-dimensional multivariate binary data with application to the analysis of microbial DNA fingerprints. Biometrics 58, 378–386].

A nonparametric plug-in rule for selecting optimal block lengths for block bootstrap methods

In this paper, we consider the problem of empirical choice of optimal block sizes for block bootstrap estimation of population parameters. We suggest a nonparametric plug-in principle that can be used for estimating ‘mean squared error’-optimal smoothing parameters in general curve estimation problems, and establish its validity for estimating optimal block sizes in various block bootstrap estimation problems. A key feature of the proposed plug-in rule is that it can be applied without explicit analytical expressions for the constants that appear in the leading terms of the optimal block lengths. Furthermore, we also discuss the computational efficacy of the method and explore its finite sample properties through a simulation study.

Improved mean shift segmentation approach for natural images

This paper proposes an improved natural image segmentation approach that is more effective, more controllable and more stable under various backgrounds than the traditional mean shift segmentation. The proposed approach employs following four new aspects: the changeable color bandwidth, the direct density searching, the global optimization for mode merging, and the elimination of texture patches. In bandwidth selection, the optimal color bandwidth under Plug-in rule used by the traditional approach is not suitable for actual vision tasks, and a changeable color bandwidth makes it easy to control the segmentation result. The performance of the direct density searching is better than that of mean shift under the same spatial bandwidth. A global optimization criterion for mode merging stabilizes the segmentation result of different images. The elimination of texture patches mostly removes the small patches resulting from texture. In addition, after mode detection, an image is partitioned into some local patches, each of which corresponds to a local mode. These patches are got with color information, and they can be taken as the initial segmentation for further processing that is based on a global optimization criterion constructed by texture features.

Estimates of error probability for complex Gaussian channels with generalized likelihood ratio detection

We derive approximate expressions for the probability of error in a two-class hypothesis testing problem in which the two hypotheses are characterized by zero-mean complex Gaussian distributions. These error expressions are given in terms of the moments of the test statistic employed and we derive these moments for both the likelihood ratio test, appropriate when class densities are known, and the generalized likelihood ratio test, appropriate when class densities must be estimated from training data. These moments are functions of class distribution parameters which are generally unknown so we develop unbiased moment estimators in terms of the training data. With these, accurate estimates of probability of error can be calculated quickly for both the optimal and plug-in rules from available training data. We present a detailed example of the behavior of these estimators and demonstrate their application to common pattern recognition problems, which include quantifying the incremental value of larger training data collections, evaluating relative geometry in data fusion from multiple sensors, and selecting a good subset of available features.

Cross-Validation and the Estimation of Conditional Probability Densities

Many practical problems, especially some connected with forecasting, require nonparametric estimation of conditional densities from mixed data. For example, given an explanatory data vector X for a prospective customer, with components that could include the customer's salary, occupation, age, sex, marital status, and address, a company might wish to estimate the density of the expenditure, Y, that could be made by that person, basing the inference on observations of (X, Y) for previous clients. Choosing appropriate smoothing parameters for this problem can be tricky, not in the least because plug-in rules take a particularly complex form in the case of mixed data. An obvious difficulty is that there exists no general formula for the optimal smoothing parameters. More insidiously, and more seriously, it can be difficult to determine which components of X are relevant to the problem of conditional inference. For example, if the jth component of X is independent of Y, then that component is irrelevant to estimating the density of Y given X, and ideally should be dropped before conducting inference. In this article we show that cross-validation overcomes these difficulties. It automatically determines which components are relevant and which are not, through assigning large smoothing parameters to the latter and consequently shrinking them toward the uniform distribution on the respective marginals. This effectively removes irrelevant components from contention, by suppressing their contribution to estimator variance; they already have very small bias, a consequence of their independence of Y. Cross-validation also yields important information about which components are relevant; the relevant components are precisely those that cross-validation has chosen to smooth in a traditional way, by assigning them smoothing parameters of conventional size. Indeed, cross-validation produces asymptotically optimal smoothing for relevant components, while eliminating irrelevant components by oversmoothing. In the problem of nonparametric estimation of a conditional density, cross-validation comes into its own as a method with no obvious peers.

Determination of the optimal number of features for quadratic discriminant analysis via the normal approximation to the discriminant distribution

Given the joint feature-label distribution, increasing the number of features always results in decreased classification error; however, this is not the case when a classifier is designed via a classification rule from sample data. Typically, for fixed sample size, the error of a designed classifier decreases and then increases as the number of features grows. The problem is especially acute when sample sizes are very small and the potential number of features is very large. To obtain a general understanding of the kinds of feature-set sizes that provide good performance for a particular classification rule, performance must be evaluated based on accurate error estimation, and hence a model-based setting for optimizing the number of features is needed. This paper treats quadratic discriminant analysis (QDA) in the case of unequal covariance matrices. For two normal class-conditional distributions, the QDA classifier is determined according to a discriminant. The standard plug-in rule estimates the discriminant from a feature-label sample to obtain an estimate of the discriminant by replacing the means and covariance matrices by their respective sample means and sample covariance matrices. The unbiasedness of these estimators assures good estimation for large samples, but not for small samples. Our goal is to find an essentially analytic method to produce an error curve as a function of the number of features so that the curve can be minimized to determine an optimal number of features. We use a normal approximation to the distribution of the estimated discriminant. Since the mean and variance of the estimated discriminant will be exact, these provide insight into how the covariance matrices affect the optimal number of features. We derive the mean and variance of the estimated discriminant and compare feature-size optimization using the normal approximation to the estimated discriminant with optimization obtained by simulating the true distribution of the estimated discriminant. Optimization via the normal approximation to the estimated discriminant provides huge computational savings in comparison to optimization via simulation of the true distribution. Feature-size optimization via the normal approximation is very accurate when the covariance matrices differ modestly. The optimal number of features based on the normal approximation will exceed the actual optimal number when there is large disagreement between the covariance matrices; however, this difference is not important because the true misclassification error using the number of features obtained from the normal approximation and the number obtained from the true distribution differ only slightly, even for significantly different covariance matrices.

Exploring the Use of Variable Bandwidth Kernel Density Estimators

Variable bandwidth kernel density estimators increase the window width at low densities and decrease it where data concentrate. This represents an improvement over the fixed bandwidth kernel density estimators. In this article, we explore the use of one implementation of a variable kernel estimator in conjunction with several rules and procedures for bandwidth selection applied to several real datasets. The considered examples permit us to state that when working with tens or a few hundreds of data observations, least-squares cross-validation bandwidth rarely produces useful estimates; with thousands of observations, this problem can be surpassed. Optimal bandwidth and biased cross-validation (BCV), in general, oversmooth multimodal densities. The Sheather–Jones plug-in rule pro-duced bandwidths that behave slightly better in this respect. The Silverman test is considered as a very sophisticated and safe procedure to estimate the number of modes in univariate distributions; however, similar results could be obtained with the Sheather–Jones rule, but at a much lower computational cost. As expected, the variable bandwidth kernel density estimates showed fewer modes than those chosen by the Silverman test, especially those distributions in which multimodality was caused by several noisy minor modes. More research on the subject is needed.

Convergence of estimated optimal inventory levels in models with probabilistic demands

The behavior of estimations of the optimal inventory level is analyzed. Two models are studied. The demands follow unknown probability distribution function. The included density functions are estimated and a plug-in rule is suggested for computing estimates of the optimal levels. Two search algorithms are proposed and compared using Monte Carlo experiments. .

Second-order asymptotic expansion for the risk in classification of curved exponential populations

This paper deals with the problem of classifying an observation into one of two curved exponential populations. The plug-in classification rule obtained by replacing unknown parameters with their bias-adjusted ML estimators in Bayes classification rule is used. The second-order asymptotic expansion with respect to the inverses of the training sample sizes for the expected regret risk is derived.

Semiparametric analysis of transformation models with censored data

SUMMARY A unified estimation procedure is proposed for the analysis of censored data using linear transformation models, which include the proportional hazards model and the proportional odds model as special cases. This procedure is easily implemented numerically and its validity does not rely on the assumption of independence between the covariates and the censoring variable. The estimator is the same as the Cox partial likelihood estimator in the case of the proportional hazards model. Moreover, the asymptotic variance of the proposed estimator has a closed form and its variance estimator is easily obtained by plug-in rules. The method is illustrated by simulation and is applied to the Veterans' Administration lung cancer data.

Sieve Bootstrap With Variable-Length Markov Chains for Stationary Categorical Time Series

A bootstrap for stationary categorical time series based on the method of sieves is studied here. The data-generating process is approximated by the so-called variable-length Markov chain (VLMC), a flexible class of Markov models that allows for parsimonious structure. Then the resampling is given by simulating from the fitted model. It is shown that for a whole class of stationary categorical time series that is more general than VLMC, the VLMC sieve has faster rate of convergence for variance estimation than the more general block bootstrap. Results are illustrated from a theoretical and empirical perspective. For the latter, a real data application about (in-) homogeneity classification of a DNA strand is also presented. Finally, the VLMC sieve scheme enjoys an implementational advantage of using the plug-in rule for bootstrapping a statistical procedure, which generally is not the case for the block method.

Comparison of Poisson Mixture Models for Count Data Clusterization

Five methods for count data clusterization based on Poisson mixture models are described. Two of them are parametric, the others are semi-parametric. The methods emlploy the plug-in Bayes classification rule. Their performance is investigated by making use of computer simulation and compared mainly by the clusterization error rate. We also apply the clusterization procedures to real count data and discuss the results.

Bootstrapping nonparametric density estimators with empirically chosen bandwidths

We examine the way in which empirical bandwidth choice affects distributional properties of nonparametric density estimators. Two bandwidth selection methods are considered in detail: local and global plug-in rules. Particular attention is focussed on whether the accuracy of distributional bootstrap approximations is appreciably influenced by using the resample version $\hat{h}*$,rather than the sample version $\hat{h}$, of an empirical bandwidth. It is shown theoretically that,in marked contrast to similar problems in more familiar settings, no general first-order theoretical improvement can be expected when using the resampling version. In the case of local plug-in rules, the inability of the bootstrap to accurately reflect biases of the components used to construct the bandwidth selector means that the bootstrap distribution of $\hat{h}*$ is unable to capture some of the main properties of the distribution of $\hat{h}$. If the second derivative component is slightly undersmoothed then some improvements are possible through using $\hat{h}*$, but they would be difficult to achieve in practice. On the other hand, for global plug-in methods, both $\hat{h}$ and $\hat{h}*$ are such good approximations to an optimal, deterministic bandwidth that the variations of either can be largely ignored, at least at a first-order level.Thus, for quite different reasons in the two cases, the computational burden of varying an empirical bandwidth across resamples is difficult to justify.

ASYMPTOTIC STABILITY OF THE OSCV SMOOTHING PARAMETER SELECTION

The smoothing parameter selection by the one-sided cross-validation (OSCV) method is completely automatic in that it does not require extra parameters estimation. Also it reduces the variability comparable to that of plug-in rules. In this paper we derive analytically the asymptotic variance of the smoothing parameter selected by OSCV. It shows the dependency of the stability on the one-sided kerenl and tells the possibility of the optimal one-sided kernel which minimizes the asymptotic variability.

Smooth discrimination analysis

Discriminant analysis for two data sets in $\mathbb{R}^d$ with probability densities $f$ and $g$ can be based on the estimation of the set $G = \{x: f(x) \geq g(x)\}$. We consider applications where it is appropriate to assume that the region $G$ has a smooth boundary or belongs to another nonparametric class of sets. In particular, this assumption makes sense if discrimination is used as a data analytic tool. Decision rules based on minimization of empirical risk over the whole class of sets and over sieves are considered. Their rates of convergence are obtained. We show that these rules achieve optimal rates for estimation of $G$ and optimal rates of convergence for Bayes risks. An interesting conclusion is that the optimal rates for Bayes risks can be very fast, in particular, faster than the “parametric” root-$n$ rate. These fast rates cannot be guaranteed for plug-in rules.

Second-order expansion for the expected regret risk in classification of one-parametric distributions

The plug-in rule is used for the classification of random observations into one of two regular one-parametric distributions. The maximum likelihood estimates of unknown parameters obtained from the stratified training sample are used. The second-order asymptotic expansion in terms of the inverses of the training sample sizes is derived for the expected regret risk. The closed-form expressions of the expansion coefficients are applicable for the performance evaluation of the proposed classification rule.