Actual Significance Levels Research Articles

The effect of Pressurised Intraperitoneal Aerosol Chemotherapy on the quality of life of patients with primary diagnosed ovarian cancer with peritoneal carcinomatosis during combined treatment

Purpose of the study. To evaluate the effect of Pressurized IntraPeritoneal Aerosol Chemotherapy (PIPAC) on the quality of life of patients during standard combined treatment of newly diagnosed ovarian cancer.Patients and methods. A combination of surgical cytoreduction with a simultaneous session of intra‑abdominal aerosol chemotherapy under pressure and subsequent systemic chemotherapy within the framework of a single hospitalization was used. The study included 164 patients (79 in the main group, 85 in the control group). All patients were operated on at the first stage, all underwent systemic cytostatic therapy according to the TS scheme (6 courses with an interval of 21 days). In the main group, 3 PIPEC sessions were added to standard treatment: the first one simultaneously with cytoreductive surgery, followed by 2 more PIPAC sessions with an interval of 42 days. To assess the quality of life, the EORTC‑QLQ – C30 questionnaire was used, which the patients filled out 4 times: before the cytoreductive stage, and then after completing the first, third and fifth courses of systemic polychemotherapy (PCT).Results. Before the start of treatment in the control group, the average general condition was 1.6 ± 1.1 points, then 1.32 ± 1.0 points, then 0.96 ± 0.97 and 0.78 ± 0.87 points at the end of the 6th course of systemic PCT. A similar situation has developed in the main group. The average baseline value before treatment was 1.7 ± 1.1 points, with the second questionnaire – 1.3 ± 1.1 points, with the third questionnaire – 0.66 ± 0.83, and 0.43 ± 0.75 points with the fourth questionnaire. In the control group, the average assessment of the quality of life before treatment was 55.0 ± 11.0 points, and then sequentially with the following three surveys 50,8 ± 7,5, 47 ± 8, 45,6 ± 8,1 points, which indicates an improvement in the quality of life against the background of ongoing treatment. In the main group, the basic level of assessment of the quality of life was 53.8 ± 11.3 points. Subsequently, an improvement in the indicators was also revealed: 49.6 ± 7.5 points, 44.2 ± 5.3 points and 42.1 ± 5.4 points, respectively. The homogeneity of the distribution in the groups was established by means of an accurate two–way Wilcoxon–Mann‑Whitney rank sum criterion, the actual significance level was p = 0.498. A two‑factor model of nonparametric analysis of variance (ANOVA) was used to analyze the data obtained. To identify a significant difference between the indicators at different stages of the study, the Page rank criterion for an ordered alternative and an algorithm based on the Friedman rank sum test were used. As a result, it was shown that the revealed positive dynamics of indicators within the groups over the entire study period is statistically significant (p < 0.0001).Conclusion. The results of the study showed that PIPAC not only does not reduce the quality of life and does not worsen the well‑being of patients, but also improves these indicators by the time of completion of combined treatment (after the 6th course of systemic PCT): in the main group, the overall ECOG condition at this stage was estimated at 0.43 ± 0.75 points, in the control group – 0.78 ± 0.87 points. The survey of patients using EORTC‑QLQ – C30 at this stage showed that in the main group the indicator was 42.1 ± 5.4 points versus 45.6 ± 8.1 points in the control group of the study.

Asymptotic Distributions of Empirical Interaction Information

Interaction Information is one of the most promising interaction strength measures with many desirable properties. However, its use for interaction detection was hindered by the fact that apart from the simple case of overall independence, asymptotic distribution of its estimate has not been known. In the paper we provide asymptotic distributions of its empirical versions which are needed for formal testing of interactions. We prove that for three-dimensional nominal vector normalized empirical interaction information converges to the normal law unless the distribution coincides with its Kirkwood approximation. In the opposite case the convergence is to the distribution of weighted centred chi square random variables. This case is of special importance as it roughly corresponds to interaction information being zero and the asymptotic distribution can be used for construction of formal tests for interaction detection. The result generalizes result in Han (Inf Control 46(1):26–45 1980) for the case when all coordinate random variables are independent. The derivation relies on studying structure of covariance matrix of asymptotic distribution and its eigenvalues. For the case of 3 × 3 × 2 contingency table corresponding to study of two interacting Single Nucleotide Polymorphisms (SNPs) for prediction of binary outcome, we provide complete description of the asymptotic law and construct approximate critical regions for testing of interactions when two SNPs are possibly dependent. We show in numerical experiments that the test based on the derived asymptotic distribution is easy to implement and yields actual significance levels consistently closer to the nominal ones than the test based on chi square reference distribution.

Size-corrected inference in fiscal policy reaction functions: a three country assessment

As a reflection of both strong persistence of debt-to-GDP ratios and correlation of respective innovations with governments’ primary surpluses, standard t-tests in policy reaction functions show actual significance levels that are up to five times larger than their nominal reference. Adopting size-controlled inference by means of Monte Carlo-based and asymptotic Bonferroni critical values, we diagnose fiscal policies in the US and the UK to be sustainable in samples covering more than 100 years. Conditioning on post-WWII subsamples and 5% nominal significance, conventional t-tests signal fiscal sustainability for these countries. In contrast, size-corrected inference hints at a lack of fiscal sustainability and, thus is recommended for the ‘real-time’ monitoring of public debt. The fiscal policy of Portugal is found to lack sustainability irrespective of the considered sample period.

Tests of Independence for a $2 \times 2$ Contingency Table with Random Margins

Fisher's exact test is commonly used for testing the hypothesis of independence between the row and column variables in a $r \times c$ contingency table. It is a ``small-sample'' test since it is used when the sample size is not large enough for the Pearsonian chi-square test to be valid. Fisher's exact test conditions on both margins of a $2 \times 2$ table leading to a hypergeometric distribution of the cell counts under independence. Moreover, it is conservative in the sense that its actual significance level falls short of the nominal level. In this paper, we modify Fisher's exact test by lifting the restriction of fixed margins and allow the margins to be random. In doing so, we propose two new tests - a likelihood ratio test in a frequentist framework and a Bayes factor test in a Bayesian framework, both of which are based on a new multinomial distributional framework. We apply the three tests on data from the Worcester Heart Attack study and compare their power functions in assessing gender difference in the therapeutic management of patients with acute myocardial infarction (AMI).

Estimating effect size when there is clustering in one treatment group.

Some experimental designs involve clustering within only one treatment group. Such designs may involve group tutoring, therapy administered by multiple therapists, or interventions administered by clinics for the treatment group, whereas the control group receives no treatment. In such cases, the data analysis often proceeds as if there were no clustering within the treatment group. A consequence is that the actual significance level of the treatment effects is larger (i.e., actual p values are larger) than nominal. Additionally, biases will be introduced in estimates of the effect sizes and their variances, leading to inflated effects and underestimated variances when clustering in the treatment group is not taken into account. These consequences of clustering can seriously compromise the interpretation of study results. This article shows how information on the intraclass correlation can be used to obtain a correction for biases in the effect sizes and their variances, and also to obtain an adjustment to the significance test for the effects of clustering.

Blinded and unblinded internal pilot study designs for clinical trials with count data

Internal pilot studies are a popular design feature to address uncertainties in the sample size calculations caused by vague information on nuisance parameters. Despite their popularity, only very recently blinded sample size reestimation procedures for trials with count data were proposed and their properties systematically investigated. Although blinded procedures are favored by regulatory authorities, practical application is somewhat limited by fears that blinded procedures are prone to bias if the treatment effect was misspecified in the planning. Here, we compare unblinded and blinded procedures with respect to bias, error rates, and sample size distribution. We find that both procedures maintain the desired power and that the unblinded procedure is slightly liberal whereas the actual significance level of the blinded procedure is close to the nominal level. Furthermore, we show that in situations where uncertainty about the assumed treatment effect exists, the blinded estimator of the control event rate is biased in contrast to the unblinded estimator, which results in differences in mean sample sizes in favor of the unblinded procedure. However, these differences are rather small compared to the deviations of the mean sample sizes from the sample size required to detect the true, but unknown effect. We demonstrate that the variation of the sample size resulting from the blinded procedure is in many practically relevant situations considerably smaller than the one of the unblinded procedures. The methods are extended to overdispersed counts using a quasi-likelihood approach and are illustrated by trials in relapsing multiple sclerosis.

Contaminated Chi-Square Modeling and Large-Scale ANOVA Testing

We propose a convenient moment-based procedure for testing the omnibus null hypothesis of no contamination of a central chi-square distribution by a non-central chi-square distribution. In sharp contrast with likelihood ratio tests for mixture models, there is no need for re-sampling or random field theory to obtain critical values. Rather, critical values are available from an asymptotic normal distribution, and there is excellent agreement between nominal and actual significance levels. This procedure may be used to model numerous chi-square statistics, obtained via monotonic transformations of F statistics, from large-scale ANOVA testing, such as that encountered in microarray data analysis. In that context, modeling chi-square statistics instead of p-values may improve detection of differential gene expression, as we demonstrate through simulation studies, while also reducing false declarations of the same, as we illustrate in a case study on aging and cognition. Our procedure may also be incorporated into a gene filtration process, which may reduce type II errors on genewise null hypotheses by justifying lighter controls for Type I errors.

Comparing Measures of Location: Some Small-Sample Results When Distributions Differ in Skewness and Kurtosis Under Heterogeneity of Variances

This article considers the problem of comparing two independent groups in terms of some measure of location. Nine methods, including a new method and a new robust estimator, were compared in terms of actual significance level and power. Simulations were performed using data generated from both theoretical distributions and data stemming from actual studies. For the theoretical distributions, the bootstrap-t B 2 t method (with 20% trimming) and Yuen's test (with 10% trimming) performed best. Otherwise, the bootstrap-t B 2 t method (with 25% trimming) and a percentile bootstrap method with trimmed means (20% and 25% trimming) or one-step M-estimator performed best.

Combined test procedures in the meta-analysis of controlled clinical trials.

Concerning the actual significance level, we investigate the commonly used combined test procedures of meta-analysis, which contain the choice of the model, in which the analysis is carried out, and the commonly used tests for treatment effect in the fixed and random effects model, and some new combined test procedures, which use an alternative test statistic in the random effects model or the t-distribution as test distribution of the commonly used test statistics. A simulation study indicates that the new combined test procedures are better with respect to a prescribed significance level.

A Comparison of Decision Methods for C pk When Data are Autocorrelated

ABSTRACT In many industrial applications, autocorrelated data are becoming increasingly common due to, for example, on-line data collection systems with high-frequency sampling. Therefore, the basic assumption of independent observations for process capability analysis is not valid. The purpose of this article is to compare decision methods using the process capability index C pk when data are autocorrelated. This is done through a case study followed by a simulation study. In the simulation study the actual significance level and power of the decision methods are investigated. The outcome of the article is that two methods appeared to be better than the others.

New heterogeneous test statistics for the unbalanced fixed-effect nested design

When the underlying variances are unknown or/and unequal, using the conventional F test is problematic in the two-factor hierarchical data structure. Prompted by the approximate test statistics (Welch and Alexander-Govern methods), the authors develop four new heterogeneous test statistics to test factor A and factor B nested within A for the unbalanced fixed-effect two-stage nested design under variance heterogeneity. The actual significance levels and statistical power of the test statistics were compared in a simulation study. The results show that the proposed procedures maintain better Type I error rate control and have greater statistical power than those obtained by the conventional F test in various conditions. Therefore, the proposed test statistics are recommended in terms of robustness and easy implementation.

Making Informed Choices about Microarray Data Analysis

Making Informed Choices about Microarray Data Analysis

The “four sigma” limit for significant periodicities

Most of the pulsations which have been observed in sdB stars have very low amplitudes. Detection of modes from photometric observations is usually performed by noting the most prominent peaks in a periodogram or, equivalently, an amplitude spectrum. A common strategy is to assume that peaks which exceed four times the mean noise level correspond to significant periodicities. The efficiency of the assumption is investigated by simulation. It is found that the actual significance levels corresponding to the four sigma limit may vary by orders of magnitude, depending on the exact data configuration.

The Significance Level of the Standard Test for a Treatment Effect in Meta-analysis

The random effects model is routinely used in meta-analysis. Although a variety of approaches have been proposed for implementing this in practice, the most popular method is that suggested by DerSimonian and Laird. The resulting inferences are not exact, however, and modifications to the methodology have been suggested. By deriving the exact distribution of the resulting test statistic, under the null hypothesis and in the simplified scenario where all studies are the same size, it is possible to obtain the actual significance level of the standard hypothesis test for a treatment effect. In particular it is found that, if there is a small number of studies and considerable between-study variation, the actual significance level can be much larger than the nominal level. This and other findings are illustrated using examples concerning the effectiveness of treatments to reduce serum cholesterol for preventing death in patients with no history of heart attacks, the use of glycerol for patients who have suffered an acute stroke, and the use of amisulpride to treat schizophrenia.

The impact of misspecification of residual error or correlation structure on the type I error rate for covariate inclusion

It has been shown that when using the FOCE method in NONMEM, the likelihood ratio test (LRT) can be sensitive to the use of an inappropriate estimation method in that ignoring an existing eta-epsilon interaction leads to actual significance levels for type I errors being higher than the nominal levels. The objective of this study was to assess through simulations the LRT sensitivity to various types of residual error model misspecifications in both continuous and categorical data. The study contained two parts, simulations based on continuous and categorical data. Data sets containing 250 individuals with up to 24 observations per individual were simulated multiple times (1000) with different types of residual error models for the continuous data and different strength of correlation between observations for the categorical data. The data sets were analyzed using either the correct or a simpler (incorrect) model with or without addition of a covariate. The type I error rate of inclusion of the non-informative covariate on the 5% level was calculated as the number of runs where the drop in the objective function value (OFV) was larger than 3.84 when the covariate relationship was included in the model using the correct or the incorrect model. The difference in OFV between the model with the correct and the incorrect structure was also calculated as a measure of the residual error model misspecification. For continuous data the FOCE method was used in most cases (with interaction when appropriate). The Laplacian estimation method was used for one of the continuous models and for categorical data. The results showed that the residual error model misspecifications when the erroneous model was used were pronounced, as indicated by the OFV being substantially higher than for the corresponding correct models. The significance levels of the LRT with the incorrect model were appropriate in all cases but ignoring (serial) correlations between observations (continuous and categorical data) as well as when the eta-epsilon interaction was ignored (which has previously been shown, continuous data). When ignoring correlation, the type I error rates were shown to be sensitive to the correlation strength, the number of observations per individual and the magnitude of the inter-individual variability on clearance. We conclude that the LRT appears robust towards all tested cases, but ignoring (serial) correlations between observations and eta-epsilon interaction.

Improved p-Value Tests for Comparing Two Independent Binomial Proportions

In the comparison of two independent binomial proportions for small or moderate sample sizes, Fisher's exact test can be conservative in the sense of its actual significance level (or size) being much less than the nominal level. Boschloo (1970) proposed a procedure, which raises the significance level used for the p-value test derived from Fisher's exact test, such that the size of the p-value test is closer to and below the nominal level. Röhmel and Mansmann (1999) used the unconditional approach, called the supremum procedure, to a valid p-value to obtain a modified p-value test, and showed that the modified p-value test is always better than the original one. In this article, we show that Röhmel and Mansmann's procedure is equivalent to Boschloo's procedure. The supremum procedure will then be applied to some well-known p-value procedures appeared in the literature. Numerical studies including two real cases are given to illustrate the advantages of the proposed modified p-value tests. Numerical results also show that the sizes of modified p-value tests are closer to nominal levels than those of the original p-value tests for many cases, especially in case of unbalanced sample sizes.

The Significance Level of Meta-Regression's Standard Hypothesis Test

Meta-regression is routinely used in the context of meta-analysis to assess the potential impact of covariates on the treatment effect. The limitations and pitfalls of this type of analysis have been documented, however, in particular, it has been claimed that the actual significance levels of the resulting tests can be higher than nominal levels. In order to examine the validity of this claim analytically, the distribution of meta-regression's standard test statistic, in the simplified scenario where the studies are the same size, is derived. The resulting significance levels can be used to give an indication of these more generally and it is found that relatively large numbers of studies are needed to provide actual significance levels that are sufficiently similar to the corresponding nominal levels. These findings can be used to inform those performing standard meta-regression hypothesis tests, so that they can avoid adopting the usual procedure in situations where this is likely to be inappropriate.

How conservative is Fisher's exact test? A quantitative evaluation of the two‐sample comparative binomial trial

The debate as to which statistical methodology is most appropriate for the analysis of the two-sample comparative binomial trial has persisted for decades. Practitioners who favor the conditional methods of Fisher, Fisher's exact test (FET), claim that only experimental outcomes containing the same amount of information should be considered when performing analyses. Hence, the total number of successes should be fixed at its observed level in hypothetical repetitions of the experiment. Using conditional methods in clinical settings can pose interpretation difficulties, since results are derived using conditional sample spaces rather than the set of all possible outcomes. Perhaps more importantly from a clinical trial design perspective, this test can be too conservative, resulting in greater resource requirements and more subjects exposed to an experimental treatment. The actual significance level attained by FET (the size of the test) has not been reported in the statistical literature. Berger (J. R. Statist. Soc. D (The Statistician) 2001; 50:79-85) proposed assessing the conservativeness of conditional methods using p-value confidence intervals. In this paper we develop a numerical algorithm that calculates the size of FET for sample sizes, n, up to 125 per group at the two-sided significance level, alpha = 0.05. Additionally, this numerical method is used to define new significance levels alpha(*) = alpha+epsilon, where epsilon is a small positive number, for each n, such that the size of the test is as close as possible to the pre-specified alpha (0.05 for the current work) without exceeding it. Lastly, a sample size and power calculation example are presented, which demonstrates the statistical advantages of implementing the adjustment to FET (using alpha(*) instead of alpha) in the two-sample comparative binomial trial.

On a Test of the Difference Between Two Binomial Proportions

The problem of testing for the equality of the proportions of two independent binomial populations is covered in the literature by a very extensive research. The most popular procedures are Fisher's exact test and the chi-square test with (or without) Yates's continuity correction. However, testing for a given difference in the proportions received very limited attention. In this article we propose two new test procedures which use hypothesis-dependent variance estimates, and investigate their performances, as well as the performances of the commonly used tests. We show that the proposed tests are more robust in that their actual significance levels are usually closer to the nominal levels than those of the commonly used tests.

Critical values for testing a unit root in finite samples from the MA(1)

This study, using a certain simulation strategy, for the exact maximum likelihood estimator of θ from the MA(1), estimates appropriate percentiles, together with their standard errors, offering a new set of critical values for testing in finite samples H0: θ = −1, against H1: θ>−1. In this way, appropriate regions for rejecting the null or being in uncertainty are defined, regardless of the values of MA(1) parameters. The new set of critical values produce both actual level of significance close to the nominal one and, when θ is not very close to −1, comparable power with the up to now suggested asymptotic values. These asymptotic values, unfortunately, lead to actual level of significance considerably greater than the nominal one, especially in large samples.