Empirical Coverage Probabilities Research Articles

On Confidence Interval Construction for Establishing Equivalence of Two Binary-Outcome Treatments in Matched-Pair Studies in the Presence of Incomplete Data

Matched-pair design is often adopted in equivalence or non-inferiority trials to increase the efficiency of binary-outcome treatment comparison. Briefly, subjects are required to participate in two binary-outcome treatments (e.g., old and new treatments via crossover design) under study. To establish the equivalence between the two treatments at the α significance level, a (1−α)100% confidence interval for the correlated proportion difference is constructed and determined if it is entirely lying in the interval (−δ0,δ0) for some clinically acceptable threshold δ0 (e.g., 0.05). Nonetheless, some subjects may not be able to go through both treatments in practice and incomplete data thus arise. In this article, a hybrid method for confidence interval construction for correlated rate difference is proposed to establish equivalence between two treatments in matched-pair studies in the presence of incomplete data. The basic idea is to recover variance estimates from readily available confidence limits for single parameters. We compare the hybrid Agresti–Coull, Wilson score and Jeffreys confidence intervals with the asymptotic Wald and score confidence intervals with respect to their empirical coverage probabilities, expected confidence widths, ratios of left non-coverage probability, and total non-coverage probability. Our simulation studies suggest that the Agresti–Coull hybrid confidence intervals is better than the score-test-based and likelihood-ratio-based confidence interval in small to moderate sample sizes in the sense that the hybrid confidence interval controls its true coverage probabilities around the pre-assigned coverage level well and yield shorter expected confidence widths. A real medical equivalence trial with incomplete data is used to illustrate the proposed methodologies.

Non‐inferiority test and confidence interval for the difference in correlated proportions in diagnostic procedures based on multiple raters

The efficacy of diagnostic procedures is generally evaluated on the basis of the results from multiple raters. However, there are few adequate methods of performing non-inferiority tests with confidence intervals to compare the accuracies (sensitivities or specificities) when multiple raters are considered. We propose new statistical methods for comparing the accuracies of two diagnostic procedures in a non-inferiority trial, on the basis of the results from multiple independent raters who are also independent of the study centers. We consider a study design in which each patient is subjected to two diagnostic procedures and all images are read by all raters. By assuming a multinomial distribution for matched-pair categorical data arising from the study design, we derive a score-based full menu, that is, a non-inferiority test, confidence interval and sample size formula, for inference of the difference in correlated proportions between the two diagnostic procedures. We conduct Monte Carlo simulation studies to examine the validity of the proposed methods, which showed that the proposed test has a size closer to the nominal significance level than a Wald-type test and that the proposed confidence interval has better empirical coverage probability than a Wald-type confidence interval. We illustrate the proposed methods with data from a study of diagnostic procedures for the diagnosis of oesophageal carcinoma infiltrating the tracheobronchial tree.

Bootstrap confidence interval for a correlation curve

A correlation curve measures the strength of the association between two variables locally at different values of x . The purpose of this study is to obtain point-wise confidence intervals for a correlation curve using wild bootstrap techniques. Empirical coverage probabilities are found to be close to the specified nominal level. Bootstrapping is an attractive alternative to confidence intervals based on asymptotic expressions that have slow rate of convergence.

Method comparison on confidence interval construction for the slope in a linear measurement model with heteroscedastic errors

In analytical chemistry, the evaluation on performance accuracy of an analytical method is an important issue. When an adjusted or new method (test method) is developed, the linear measurement error model is commonly used to compare it with another reference method. For this routine practice, the measurements on the reference method can be placed on the x‐axis, whereas those of the test method on the y‐axis, then the slope of this linear relationship indicates the agreement between them and also the performance of the test method. Under the assumption that both variables are subject to heteroscedastic measurement errors, a novel approach based on the concepts of a generalized pivotal quantity (GPQ) is proposed to construct confidence intervals for the slope. Its performance is compared with two maximum likelihood estimation (MLE)‐based approaches through simulation studies. It is shown that the proposed GPQ‐based approach is capable of maintaining the empirical coverage probabilities close to the nominal level and yielding reasonable expected lengths. The GPQ‐based approach can be recommended in practical use because of its easy implementation and better performance than the MLE‐based approaches in most simulation scenarios. Two real datasets are given to illustrate the approaches. Copyright © 2011 John Wiley & Sons, Ltd.

Nonparametric time series forecasting with dynamic updating

We present a nonparametric method to forecast a seasonal univariate time series, and propose four dynamic updating methods to improve point forecast accuracy. Our methods consider a seasonal univariate time series as a functional time series. We propose first to reduce the dimensionality by applying functional principal component analysis to the historical observations, and then to use univariate time series forecasting and functional principal component regression techniques. When data in the most recent year are partially observed, we improve point forecast accuracy by using dynamic updating methods. We also introduce a nonparametric approach to construct prediction intervals of updated forecasts, and compare the empirical coverage probability with an existing parametric method. Our approaches are data-driven and computationally fast, and hence they are feasible to be applied in real time high frequency dynamic updating. The methods are demonstrated using monthly sea surface temperatures from 1950 to 2008.

Asymptotic confidence interval construction for proportion difference in medical studies with bilateral data

Bilateral dichotomous data are very common in modern medical comparative studies (e.g. comparison of two treatments in ophthalmologic, orthopaedic and otolaryngologic studies) in which information involving paired organs (e.g. eyes, ears and hips) is available from each subject. In this article, we study various confidence interval estimators for proportion difference based on Wald-type statistics, Fieller theorem, likelihood ratio statistic, score statistics and bootstrap resampling method under the dependence or/and independence models for bilateral binary data. Performance is evaluated with respect to the coverage probability and expected width via simulation studies. Our empirical results show that (1) ignoring the dependence feature of bilateral data could lead to severely incorrect coverage probabilities; and (2) Wald-type, score-type and bootstrap confidence intervals based on the dependence model perform satisfactorily for small to large sample sizes in the sense that their empirical coverage probabilities are close to the pre-specified nominal confidence level and are hence recommended. A real data from an otolaryngologic study is used to illustrate the proposed methods.

A comparison of methods for the construction of confidence interval for relative risk in stratified matched‐pair designs

A stratified matched-pair study is often designed for adjusting a confounding effect or effect of different trails/centers/ groups in modern medical studies. The relative risk is one of the most frequently used indices in comparing efficiency of two treatments in clinical trials. In this paper, we propose seven confidence interval estimators for the common relative risk and three simultaneous confidence interval estimators for the relative risks in stratified matched-pair designs. The performance of the proposed methods is evaluated with respect to their type I error rates, powers, coverage probabilities, and expected widths. Our empirical results show that the percentile bootstrap confidence interval and bootstrap-resampling-based Bonferroni simultaneous confidence interval behave satisfactorily for small to large sample sizes in the sense that (i) their empirical coverage probabilities can be well controlled around the pre-specified nominal confidence level with reasonably shorter confidence widths; and (ii) the empirical type I error rates of their associated test statistics are generally closer to the pre-specified nominal level with larger powers. They are hence recommended. Two real examples from clinical laboratory studies are used to illustrate the proposed methodologies.

Confidence Interval for Shrinkage Parameters in Ridge Regression

In ridge regression, the estimation of ridge parameter k is an important problem. There are several methods available in the literature to do this job some what efficiently. However, no attempts were made to suggest a confidence interval for the ridge parameter using the knwoledge from the data. In this article, we propose a data dependent confidence interval for the ridge parameter k. The method of obtaining the confidence interval is illustrated with the help of a data set. A simulation study indicates that the empirical coverage probability of the suggested confidence intervals are quite high.

Forecasting functional time series

We propose forecasting functional time series using weighted functional principal component regression and weighted functional partial least squares regression. These approaches allow for smooth functions, assign higher weights to more recent data, and provide a modeling scheme that is easily adapted to allow for constraints and other information. We illustrate our approaches using age-specific French female mortality rates from 1816 to 2006 and age-specific Australian fertility rates from 1921 to 2006, and show that these weighted methods improve forecast accuracy in comparison to their unweighted counterparts. We also propose two new bootstrap methods to construct prediction intervals, and evaluate and compare their empirical coverage probabilities.

Complementary Log–Log Regression for the Estimation of Covariate‐Adjusted Prevalence Ratios in the Analysis of Data from Cross‐Sectional Studies

We assessed complementary log-log (CLL) regression as an alternative statistical model for estimating multivariable-adjusted prevalence ratios (PR) and their confidence intervals. Using the delta method, we derived an expression for approximating the variance of the PR estimated using CLL regression. Then, using simulated data, we examined the performance of CLL regression in terms of the accuracy of the PR estimates, the width of the confidence intervals, and the empirical coverage probability, and compared it with results obtained from log-binomial regression and stratified Mantel-Haenszel analysis. Within the range of values of our simulated data, CLL regression performed well, with only slight bias of point estimates of the PR and good confidence interval coverage. In addition, and importantly, the computational algorithm did not have the convergence problems occasionally exhibited by log-binomial regression. The technique is easy to implement in SAS (SAS Institute, Cary, NC), and it does not have the theoretical and practical issues associated with competing approaches. CLL regression is an alternative method of binomial regression that warrants further assessment.

Wavelet-based confidence intervals for the self-similarity parameter

We propose and compare several methods of constructing wavelet-based confidence intervals for the self-similarity parameter in heavy-tailed observations. We use empirical coverage probabilities to assess the procedures by applying them to Linear Fractional Stable Motion with many choices of parameters. We find that the asymptotic confidence intervals provide empirical coverage often much lower than nominal. We recommend the use of resampling confidence intervals. We also propose a procedure for monitoring the constancy of the self-similarity parameter and apply it to Ethernet data sets.

Efficient interval estimation for age-adjusted cancer rates

The age-adjusted cancer rates are defined as the weighted average of the age-specific cancer rates, where the weights are positive, known, and normalized so that their sum is 1. Fay and Feuer developed a confidence interval for a single age-adjusted rate based on the gamma approximation. Fay used the gamma approximations to construct an F interval for the ratio of two age-adjusted rates. Modifications of the gamma and F intervals are proposed and a simulation study is carried out to show that these modified gamma and modified F intervals are more efficient than the gamma and F intervals, respectively, in the sense that the proposed intervals have empirical coverage probabilities less than or equal to their counterparts, and that they also retain the nominal level. The normal and beta confidence intervals for a single age-adjusted rate are also provided, but they are shown to be slightly liberal. Finally, for comparing two correlated age-adjusted rates, the confidence intervals for the difference and for the ratio of the two age-adjusted rates are derived incorporating the correlation between the two rates. The proposed gamma and F intervals and the normal intervals for the correlated age-adjusted rates are recommended to be implemented in the Surveillance, Epidemiology and End Results Program of the National Cancer Institute.

Performance evaluation of recent procedures for steady-state simulation analysis

The performance of the batch-means procedure ASAP3 and the spectral procedure WASSP is evaluated on test problems with characteristics typical of practical applications of steady-state simulation analysis procedures. ASAP3 and WASSP are sequential procedures designed to produce a confidence-interval estimator for the mean response that satisfies user-specified half-length and coverage-probability requirements. ASAP3 is based on an inverse Cornish-Fisher expansion for the classical batch-means t-ratio, whereas WASSP is based on a wavelet estimator of the batch-means power spectrum. Regarding closeness of the empirical coverage probability and average half-length of the delivered confidence intervals to their respective nominal levels, both procedures compared favorably with the Law-Carson procedure and the original ASAP algorithm. Regarding the average sample sizes required for decreasing levels of maximum confidence-interval half-length, ASAP3 and WASSP exhibited reasonable efficiency in the test problems.

Evaluating predictive performance of value-at-risk models in emerging markets: a reality check

We investigate the predictive performance of various classes of value-at-risk (VaR) models in several dimensions—unfiltered versus filtered VaR models, parametric versus nonparametric distributions, conventional versus extreme value distributions, and quantile regression versus inverting the conditional distribution function. By using the reality check test of White (2000), we compare the predictive power of alternative VaR models in terms of the empirical coverage probability and the predictive quantile loss for the stock markets of five Asian economies that suffered from the 1997–1998 financial crisis. The results based on these two criteria are largely compatible and indicate some empirical regularities of risk forecasts. The Riskmetrics model behaves reasonably well in tranquil periods, while some extreme value theory (EVT)-based models do better in the crisis period. Filtering often appears to be useful for some models, particularly for the EVT models, though it could be harmful for some other models. The CaViaR quantile regression models of Engle and Manganelli (2004) have shown some success in predicting the VaR risk measure for various periods, generally more stable than those that invert a distribution function. Overall, the forecasting performance of the VaR models considered varies over the three periods before, during and after the crisis. Copyright © 2006 John Wiley & Sons, Ltd.

Statistical methods for multivariate interval‐censored recurrent events

Multi-type recurrent event data arise when two or more different kinds of events may occur repeatedly over a period of observation. The scientific objectives in such settings are often to describe features of the marginal processes and to study the association between the different types of events. Interval-censored multi-type recurrent event data arise when the precise event times are unobserved, but intervals are available during which the events are known to have occurred. This type of data is common in studies of patients with advanced cancer, for example, where the events may represent the development of different types of metastatic lesions which are only detectable by conducting bone scans of the entire skeleton. In this setting it is of interest to characterize the incidence of the various types of bone lesions, to estimate the impact of treatment and other covariate effects on the development of new lesions, and to understand the relationship between the processes generating the bone lesions. We develop joint models for multi-type interval-censored recurrent events which accommodate dependencies between different types of events and enable one to examine the covariate effects via regression. However, since the marginal likelihood resulting from the multivariate random effect model is intractable, we describe a Gibbs sampling algorithm to facilitate model fitting and inference. We use generalized estimating equations for estimation and inference based on marginal models. The finite sample properties of the marginal approach are studied via simulation. The estimates of both the regression coefficients and the variance-covariance parameters are shown to have negligible bias and 95 per cent confidence intervals based on the asymptotic variance formula are shown to have excellent empirical coverage probabilities in all of the settings considered. The application of these methods to data from a trial of women with advanced breast cancer provides insight into the clinical course of bone metastases in this population.

Marginal analysis of incomplete longitudinal binary data: a cautionary note on LOCF imputation.

In recent years there has been considerable research devoted to the development of methods for the analysis of incomplete data in longitudinal studies. Despite these advances, the methods used in practice have changed relatively little, particularly in the reporting of pharmaceutical trials. In this setting, perhaps the most widely adopted strategy for dealing with incomplete longitudinal data is imputation by the "last observation carried forward" (LOCF) approach, in which values for missing responses are imputed using observations from the most recently completed assessment. We examine the asymptotic and empirical bias, the empirical type I error rate, and the empirical coverage probability associated with estimators and tests of treatment effect based on the LOCF imputation strategy. We consider a setting involving longitudinal binary data with longitudinal analyses based on generalized estimating equations, and an analysis based simply on the response at the end of the scheduled follow-up. We find that for both of these approaches, imputation by LOCF can lead to substantial biases in estimators of treatment effects, the type I error rates of associated tests can be greatly inflated, and the coverage probability can be far from the nominal level. Alternative analyses based on all available data lead to estimators with comparatively small bias, and inverse probability weighted analyses yield consistent estimators subject to correct specification of the missing data process. We illustrate the differences between various methods of dealing with drop-outs using data from a study of smoking behavior.

A simulation study on the accuracy of position and effect estimates of linked QTL and their asymptotic standard deviations using multiple interval mapping in an F2 scheme

Approaches like multiple interval mapping using a multiple-QTL model for simultaneously mapping QTL can aid the identification of multiple QTL, improve the precision of estimating QTL positions and effects, and are able to identify patterns and individual elements of QTL epistasis. Because of the statistical problems in analytically deriving the standard errors and the distributional form of the estimates and because the use of resampling techniques is not feasible for several linked QTL, there is the need to perform large-scale simulation studies in order to evaluate the accuracy of multiple interval mapping for linked QTL and to assess confidence intervals based on the standard statistical theory. From our simulation study it can be concluded that in comparison with a monogenetic background a reliable and accurate estimation of QTL positions and QTL effects of multiple QTL in a linkage group requires much more information from the data. The reduction of the marker interval size from 10 cM to 5 cM led to a higher power in QTL detection and to a remarkable improvement of the QTL position as well as the QTL effect estimates. This is different from the findings for (single) interval mapping. The empirical standard deviations of the genetic effect estimates were generally large and they were the largest for the epistatic effects. These of the dominance effects were larger than those of the additive effects. The asymptotic standard deviation of the position estimates was not a good criterion for the accuracy of the position estimates and confidence intervals based on the standard statistical theory had a clearly smaller empirical coverage probability as compared to the nominal probability. Furthermore the asymptotic standard deviation of the additive, dominance and epistatic effects did not reflect the empirical standard deviations of the estimates very well, when the relative QTL variance was smaller/equal to 0.5. The implications of the above findings are discussed.

Confidence Intervals and Accuracy Estimation for Heavy-Tailed Generalized Pareto Distributions

The generalized Pareto distribution (GPD) is a two-parameter family of distributions which can be used to model exceedances over a threshold. We compare the empirical coverage of some standard bootstrap and likelihood-based confidence intervals for the parameters and upper p-quantiles of the GPD. Simulation results indicate that none of the bootstrap methods give satisfactory intervals for small sample sizes. By applying a general method of D. N. Lawley, correction factors for likelihood ratio statistics of parameters and quantiles of the GPD have been calculated. Simulations show that for small sample sizes accuracy of confidence intervals can be improved by incorporating the computed correction factors to the likelihood-based confidence intervals. While the modified likelihood method has better empirical coverage probability, the mean length of produced intervals are not longer than corresponding bootstrap confidence intervals. This article also investigates the performance of some bootstrap methods for estimation of accuracy measures of maximum likelihood estimators of parameters and quantiles of the GPD.

On Constructing Confidence Intervals for a Standardized Mean Difference in Meta-analysis

Under the random effects model for meta-analysis, confidence intervals for the overall effect are typically constructed using quantiles of the standard normal distribution. We discuss confidence intervals based on both the standard normal distribution and the t-distribution, in conjunction with several methods of estimating the heterogeneity variance for a standardized mean difference, and we compare the empirical coverage probabilities of the intervals using simulation. The coverage probabilities of intervals based on an approximate t-statistic are higher than the coverage probabilities for the standard normal intervals, and are very close to the specified confidence level even for small meta-analysis sample size. Moreover, intervals based on the approximate t-statistic appear relatively robust to different methods of estimating the heterogeneity variance, unlike the normal intervals. Thus, we conclude that confidence intervals based on the t-statistic are superior to the standard normal confidence intervals for a standardized mean difference, and should be used by practitioners in place of the normal intervals.

Evaluating Normal Approximation Confidence Intervals for Measures of 2 × 2 Association with Applications to Twin Data

AbstractTwin data are of interest to genetic epidemiologists for exploring the underlying genetic basis of disease development. When the outcome is binary, several indices of 2 × 2 association can be used to measure the degree of within twin similarity. All such measures share a common feature, in that they can be expressed as a monotonic increasing function of the within twin correlation. The sampling distributions of their estimates are influenced by the sample size, the correlation and the marginal distribution of the binary response. In this paper we use Monte‐Carlo simulations to estimate the empirical coverage probabilities and evaluate the adequacy of the classical normal confidence intervals on the population values of these measures.