Empirical Significance Levels Research Articles

Testing a Unit Root Based on Aggregate Time Series

Many time series encountered in practice are nonstationary, and instead are often generated from a process with a unit root. Because of the process of data collection or the practice of researchers, time series used in analysis and modeling are frequently obtained through temporal aggregation. As a result, the series used in testing for a unit root are often time series aggregates. In this paper, we study the effects of the use of aggregate time series on the Dickey–Fuller test for a unit root. We start by deriving a proper model for the aggregate series. Based on this model, we find the limiting distributions of the test statistics and illustrate how the tests are affected by the use of aggregate time series. The results show that those distributions shift to the right and that this effect increases with the order of aggregation, causing a strong impact both on the empirical significance level and on the power of the test. To correct this problem, we present tables of critical points appropriate for the tests based on aggregate time series and demonstrate their adequacy. Examples illustrate the conclusions of our analysis.

Empirical significance values for linkage analysis: trait simulation using posterior model distributions from MCMC oligogenic segregation analysis

Variance-components (VC) linkage analysis is a powerful model-free method for assessing linkage, but the distribution of VC logarithm of the odds ratio (LOD) scores may deviate substantially from the assumed asymptotic distribution. Typically, the null distribution of the VC-LOD score and other linkage statistics has been estimated by generating new genotype data independently of the trait data, and computing a linkage statistic for many such marker-simulated data sets. However, marker simulation is susceptible to errors in the assumed marker and map model and is computationally intensive. Here, we describe a method for generating posterior distributions of linkage statistics through simulation of trait data based on the original sample and on results from an initial scan using a Bayesian Markov-chain Monte Carlo (MCMC) approach for oligogenic segregation analysis. We use samples of oligogenic trait models taken from the posterior distribution to generate new samples of trait data, which were paired with the original marker data for analysis. Empirical P-values obtained from trait and marker simulation were similar when derived for several strong linkage signals from published linkage scans, and for analysis of data with a known, simulated, trait model. Furthermore, trait simulation produces the expected null distribution of VC-LOD scores and is computationally fast when marker identity-by-descent estimates from the original data could be reused. These results suggest that trait simulation gives valid estimates of statistical significance of linkage signals. Finally, these results also demonstrate the feasibility of obtaining empirical significance levels for evaluating Bayesian oligogenic linkage signals with either marker or trait simulation.

Genomewide scan in Ashkenazi Jewish families demonstrates evidence of linkage of ocular refraction to a QTL on chromosome 1p36

The development of refractive error is mediated by both environmental and genetic factors. We performed regression-based quantitative trait locus (QTL) linkage analysis on Ashkenazi Jewish families to identify regions in the genome responsible for ocular refraction. We measured refractive error on individuals in 49 multi-generational American families of Ashkenazi Jewish descent. The average family size was 11.1 individuals and was composed of 2.7 generations. Recruitment criteria specified that each family contain at least two myopic members. The mean spherical equivalent refractive error in the sample was -3.46D (SD=3.29) and 87% of individuals were myopic. Microsatellite genotyping with 387 markers was performed on 411 individuals. We performed multipoint regression-based linkage analysis for ocular refraction and a log transformation of the trait using the statistical package Merlin-Regress. Empirical genomewide significance levels were estimated through gene-dropping simulations by generating random genotypes at each of the 387 markers in 200 replicates of our pedigrees. Maximum LOD scores of 9.5 for ocular refraction and 8.7 for log-transformed refraction (LTR) were observed at 49.1 cM on chromosome 1p36 between markers D1S552 and D1S1622. The empirical genomewide significance levels were P=0.065 for ocular refraction and P<0.005 for LTR, providing strong evidence for linkage of refraction to this locus. The inter-marker region containing the peak spans 11 Mb and contains approximately 189 genes. We found genomewide significant evidence for linkage of refractive error to a novel QTL on chromosome 1p36 in an Ashkenazi Jewish population.

Association analysis of estrogen receptor α gene polymorphisms with cross-sectional geometry of the femoral neck in Caucasian nuclear families

Bone geometry is a key factor in bone strength, which is the ultimate intrinsic determinant of fracture risk. Though the heritability of bone geometry is high, little effort has been spent on searching for the underlying genes. In this study, employing a sample of 1,873 subjects from 405 Caucasian nuclear families, we studied seven single nucleotide polymorphisms (SNPs) and their haplotypes of the ER-alpha gene for association with six hip geometric variables, namely, cross-sectional area (CSA), cortical thickness (CT), endocortical diameter (ED), subperiosteal width (W), sectional modulus (Z) and buckling ratio (BR). The major method used was the quantitative transmission disequilibrium test (QTDT). Our major findings were summarized below. The within-family association between SNP4 (rs1801132) in exon 4 with endocortical diameter and subperiosteal width was detected in single locus analyses (P=0.008 and 0.021, respectively) and verified in haplotype analyses (P=0.034 and 0.058, respectively). The total association of SNP4 with these two diameters was also observed in both single locus and haplotype analyses (P=0.005 and 0.031 for ED, plus P=0.003 and 0.070 for W). In addition, the total association between SNP5 (rs932477) in intron 4 with cortical thickness and buckling ratio was detected (single locus analyses: P=0.035 and 0.041, respectively). Haplotype analyses further supported the above association (P=0.010 and 0.004, respectively). Similar patterns of associations with the studied SNPs and their haplotypes were present in subsamples stratified by sex, too. However, after permutation tests, the empirical significance level was set as P<0.011, which renders most associations insignificant. Therefore, we concluded that polymorphisms in the ER-alpha gene were nominally associated with femoral neck (FN) geometry variables estimated from DXA. Such genetic effects on hip geometry were not sex specific.

Establishing appropriate genome-wide significance levels for canine linkage analyses.

A threshold of 3.3 for a genome-wide maximum LOD score (MAXLOD) has been demonstrated in human linkage studies as corresponding to a type I error rate of 5%. Generalization of this work to other species assumes the presence of an infinitely dense marker map. While this assumption is increasingly realistic for the human genome, it may be unrealistic for the dog genome. In this study we establish the analytic and empirical thresholds for MAXLOD in canine linkage studies corresponding to type I error rates of 5% and 1% for autosomal traits. Empirical thresholds are computed via simulation assuming a 10 cM map with no fine mapping performed. Pedigree structures for simulations were drawn from two canine disease studies. Five thousand replicates of genome-wide null genotype data were simulated and analyzed for each disease. We determined that MAXLOD thresholds of 3.2 and 2.7 correspond to analytic and empirical type I error rates of 5%, respectively. In all cases, the MAXLOD thresholds from simulations were always at least 0.5 LOD units below the corresponding analytic thresholds. We therefore recommend that a threshold of 3.2 be used for canine linkage studies when fine mapping is performed, and that researchers perform their own simulation studies to assess genome-wide empirical significance levels when no fine mapping is performed.

Cluster Randomization Trials: A Simulation Study

A simulation study is conducted to compare several methods that test the common log odds ratio in multiple 2 × 2 tables when the data are correlated within clusters. Allowing cluster size to vary within each table, we evaluate the unadjusted Mantel-Haenszel chi-square statistic (χ2MH), the adjusted Mantel-Haenszel chi-square statistics of Rao and Scott using both an unpooled design effect (χ2RSN) and a pooled design effect (χ2RSP), the adjusted Mantel-Haenszel chi-square statistic of Donald and Donner (χ2DD), the chi-square statistic using the GEE approach (χ2GEE), the adjusted Mantel-Haenszel chi-square statistic of Begg (χ2B), the Wald (χ2W), the robust Wald (χ2RW), the score (χ2S), the robust score (χ2RS), and the adjusted Mantel-Haenszel chi-square statistics of Zhang and Boos (χ2ZBP and χ2ZBN). The test statistics above are compared in terms of empirical significance levels and empirical power levels. The robust score statistic χ2RS and the adjusted Mantel-Haenszel chi-square statistics of Zhang and Boos (χ2ZBP and χ2ZBN) generally have empirical significance levels closer to the nominal value than the other statistics. These three statistics have similar empirical power levels when the intracluster correlation is zero or the cluster sizes are balanced. χ2RS performs better in terms of empirical power levels when a positive intracluster correlation exists in the imbalance setting.

ROBUSTNESS OF PROCEDURES FOR THE BEHRENS-FISHER PROBLEMS: EXTENSION TO BIVARIATE NORMAL MIXTURES

In the applied sciences, it is often important to be able to compare the mean values of two populations. However, testing a hypothesis can be complex, if the two populations are heteroscedastic and exhibit non-normality in the data. This paper reviews currently available strategies for the multivariate Behrens-Fisher problem. It then carries out Monte Carlo comparisons of selected procedures to assess their robustness when applied to data from normal mixture distributions. The overall conclusion is that Johansen's procedure appears to work best for small sample data both in terms of empirical power and significance level. Johansen's procedure works reasonably well even with mixture data. The simulation also provides researchers with specific guidelines to follow at the early designing and planning stages of the investigation.

COMPARING EMPIRICAL DISTRIBUTIONS OF P-VALUES FROM SIMULATIONS

In the context of hypothesis testing simulation studies, this paper advocates using graphical and numerical tools to summarize the results, beyond the conventional practice of just reporting empirical levels. Some of these tools are defined by us. They are mainly based on the computation of distances between the empirical distribution function of the p-values derived from the simulation experiment and the distribution function of the U([0, 1]) random variable. Their null distribution is tabulated. A joint study of several distances reveals that important aspects of a test can pass unnoticed if only empirical significance levels are calculated. The proposed tools are applied to two practical examples, demonstrating their usefulness in the discrimination between alternative test procedures and also in the detection of data not according to the null hypothesis.

Efficient Multipoint Linkage Analysis through Reduction of Inheritance Space

Computational constraints currently limit exact multipoint linkage analysis to pedigrees of moderate size. We introduce new algorithms that allow analysis of larger pedigrees by reducing the time and memory requirements of the computation. We use the observed pedigree genotypes to reduce the number of inheritance patterns that need to be considered. The algorithms are implemented in a new version (version 2.1) of the software package GENEHUNTER. Performance gains depend on marker heterozygosity and on the number of pedigree members available for genotyping, but typically are 10-1,000-fold, compared with the performance of the previous release (version 2.0). As a result, families with up to 30 bits of inheritance information have been analyzed, and further increases in family size are feasible. In addition to computation of linkage statistics and haplotype determination, GENEHUNTER can also perform single-locus and multilocus transmission/disequilibrium tests. We describe and implement a set of permutation tests that allow determination of empirical significance levels in the presence of linkage disequilibrium among marker loci.

Two approaches for consolidating results from genome scans of complex traits: selection methods and scan statistics.

This work has two purposes: (i) empirically selecting levels of significance that maximize the fraction of markers close to a gene (hit rate) when performing linkage analyses of simulated data and (ii) evaluating the utility of a previously reported scan statistic on the same data. Genotype data were simulated from a trait model of seven susceptibility genes. For purpose (i), five statistics were evaluated on all marker loci in fifty replicates; two-point lod and heterogeneity lod scores maximized over dominance (mlod, mhlod), a multi-allelic TDT test, an affected sib-pair test (ASP), and a model-free test on all sib-pairs (ALL_SIBS). Within each replicate the fraction of markers (hit rate) significant at specified levels of significance and also (a) within fifty markers of, or (b) on the same chromosome as a major gene was calculated. For purpose (ii), scan statistics of length 15 were calculated for each chromosome and their empirical significance levels estimated on the basis of 500 replicates generated under no linkage. The scan statistic was applied to the mhlod scores from one replicate (Replicate 5). Empirical p-values for the scan statistic were determined by computing mhlod scores on 500 replicates of simulated null data. For purpose (i), significance levels between 0.001 and 0.01 had the greatest hit rate for all five methods and both criteria. For criterion (a) at the 0.001 level of significance, both mlod and mhlod displayed the highest hit rates, approximately 0.4 for each. For criterion (b), all methods but ALL_SIBS and ASP had hit rates ranging between 0.4 and 0.5. For purpose (ii), the scan statistic proved equally or more powerful than the single-locus statistic for two of the seven susceptibility genes while the remaining five genes were not detected.

Two Approaches for Consolidating Results from Genome Scans of Complex Traits: Selection Methods and Scan Statistics

This work has two purposes: (i) empirically selecting levels of significance that maximize the fraction of markers close to a gene (hit rate) when performing linkage analyses of simulated data and (ii) evaluating the utility of a previously reported scan statistic on the same data. Genotype data were simulated from a trait model of seven susceptibility genes. For purpose (i), five statistics were evaluated on all marker loci in fifty replicates; two‐point lod and heterogeneity lod scores maximized over dominance (mlod, mhlod), a multi‐allelic TDT test, an affected sib‐pair test (ASP), and a model‐free test on all sib‐pairs (ALL_SIBS). Within each replicate the fraction of markers (hit rate) significant at specified levels of significance and also (a) within fifty markers of, or (b) on the same chromosome as a major gene was calculated. For purpose (ii), scan statistics of length 15 were calculated for each chromosome and their empirical significance levels estimated on the basis of 500 replicates generated under no linkage. The scan statistic was applied to the mhlod scores from one replicate (Replicate 5). Empirical p‐values for the scan statistic were determined by computing mhlod scores on 500 replicates of simulated null data. For purpose (i), significance levels between 0.001 and 0.01 had the greatest hit rate for all five methods and both criteria. For criterion (a) at the 0.001 level of significance, both mlod and mhlod displayed the highest hit rates, approximately 0.4 for each. For criterion (b), all methods but ALL_SIBS and ASP had hit rates ranging between 0.4 and 0.5. For purpose (ii), the scan statistic proved equally or more powerful than the single‐locus statistic for two of the seven susceptibility genes while the remaining five genes were not detected. © 2001 Wiley‐Liss, Inc.

Nonlinear adjustment, long-run equilibrium and exchange rate fundamentals

We estimate nonlinear time-series models of the deviations of the dollar–sterling and dollar–mark exchange rates from the level suggested by simple monetary fundamentals over the recent floating rate period. The estimated parameters are statistically significant on the basis of empirical significance levels generated by Monte Carlo methods and imply nonlinear mean reversion towards the monetary fundamental equilibrium such that the speed of mean reversion increases with the size of the deviation from equilibrium. The results are used to provide an illustration of the overvaluation and undervaluation of the dollar against sterling and the mark over the recent float.

NONPARAMETRIC TEST PERFORMANCE FOR TRENDS IN WATER QUALITY WITH SAMPLING DESIGN APPLICATIONS1

ABSTRACT: In this paper four nonparametric tests for monotonic trend detection are compared with respect to their power and accuracy. The importance of comparing powers at equal empirical significance levels rather than nominal levels is stressed. Therefore, an appropriate graphical method is presented. The effect of the sampling frequency is also assessed using Monte Carlo simulations and a trajectory representation that visualizes the dynamics of the trade‐off between the type I and type II errors. These methods are applied to compare four nonparametrical tests (seasonal Mann. Kendall, modified seasonal Mann‐Kendall, covariance eigenvalue and covariance inversion) under several conditions. It is concluded with respect to the power that it is not worthwhile for the modified seasonal Mann‐Kendall test applied to the AR(1) process considered in this paper to increase the sampling frequency from monthly to biweekly for detecting a monotonic trend of 5 percent, 10 percent, or 15 percent of the process variance. Under these conditions the seasonal Mann‐Kendall test is highly liberal, while the covariance inversion and the covariance eigenvalue test are conservative. This research is situated in the development of an efficient sampling design for the Flemish water quality monitoring network.

A modified Mann-Kendall trend test for autocorrelated data

One of the commonly used tools for detecting changes in climatic and hydrologic time series is trend analysis. A number of statistical tests exist to assess the significance of trends in time series. One of the commonly used non-parametric trend tests is the Mann-Kendall trend test. The null hypothesis in the Mann-Kendall test is that the data are independent and randomly ordered. However, the existence of positive autocorrelation in the data increases the probability of detecting trends when actually none exist, and vice versa. Although this is a well-known fact, few studies have addressed this issue, and autocorrelation in the data is often ignored. In this study, the effect of autocorrelation on the variance of the Mann-Kendall trend test statistic is discussed. A theoretical relationship is derived to calculate the variance of the Mann-Kendall test statistic for autocorrelated data. The special cases of AR(1) and MA(1) dependence are discussed as examples. An approximation to the theoretical relationship is also presented in order to reduce computation time for long time series. Based on the modified value of the variance of the Mann-Kendall trend test statistic, a modified non-parametric trend test which is suitable for autocorrelated data is proposed. The accuracy of the modified test in terms of its empirical significance level was found to be superior to that of the original Mann-Kendall trend test without any loss of power. The modified test is applied to rainfall as well as streamflow data to demonstrate its performance as compared to the original Mann-Kendall trend test.

13 A unified family of group sequential designs

s 49s 13 A UNIFIED FAMILY OF GROUP SEQUENTIAL DESIGNS Scott S. Emerson and John M. Kittelson University of Washington Seattle, Washington It is increasingly standard practice to incorporate a formal stopping rule in the design of a clinical trial. Failure to consider the sequential nature of such hypothesis testing can inflate the type I statistical error above the nominal level. A number of group sequential designs which control the type I error have been proposed in the statistical literature. Most of these designs extend fixed sample (nonsequential) hypothesis tests by generalizing a single characteristic of a fixed sample test. The end result has been the proliferation of families of group sequential tests having only limited flexibility within a family to address the many issues which need to be considered when choosing a stopping rule. In this talk, we present a large family of group sequential test designs that includes many of the previously proposed families. Because this family is parameterized such that a user may move continuously among those designs, we believe that the use of this family in software packages will greatly facilitate the process of designing a clinical trial. We use as an example the history of the design of a large multicenter clinical trial of a new treatment for gram negative sepsis, and we illustrate the various steps of selecting a design using S-Plus. 14 CLINICAL EQUIVALENCE TEST AND CONFIDENCE INTERVAL FOR THE DIFFERENCE IN PROPORTIONS FOR THE PAIRED-SAMPLE DESIGN Toshiro Tango The Institute of Public Health Tokyo, Japan This paper considers the model for the difference between two proportions in a paired or matched design of cliical trials. This model includes a parameter indicating both inter-patients variability of response probabilities and their correlation. Under the proposed model, we derive a one-sided test for clinical equivalence based upon the efficient score: H~:~N=zs-A,H~:xN>~S-A Further, a score-based confidence interval for the difference of two proportions, IFN xs is derived. Especially, one of the merits of these methods is the applicability to 2x2 table data with off-diagonal zero cells. Monte Carlo simulation study shows that the proposed test is shown to have empirical significance levels closer to nominal a-level than the other tests recently proposed and further that the proposed confidence interval is shown to have better empirical coverage probability compared with those of the published methods. To illustrate our proposed methods, let us consider the following 2x2 data tables having zero off-diagonal cells:

Empirical genomewide significance levels established by whole genome simulations.

The advent of high-resolution genetic maps and semiautomated genotyping technology has opened the way for genome screening in genetically complex traits. Many such screens are now under way, or completed, most using multipoint nonparametric linkage analysis of affected sibling pairs. This type of linkage analysis uses all the available genotype information to calculate the maximum lod score (MLS) value at each point in the genome, and thereby generates MLS profiles along each chromosome. Any positive MLS values indicate potential linkage, but the peaks in these profiles, which may be referred to as "hits," identify the most likely locations of disease susceptibility genes. However, such analysis presents serious problems of multiple testing, and the assessment of the statistical significance of hits has become a contentious issue [Lander and Kruglyak (1995) Nat Genet 11:241-247; Curtis (1996) Nat Genet 12:356-357; Witte et al. (1996) Nat Genet 12:355-356]. Having recently completed a genome screen in multiple sclerosis, we decided to investigate the statistical properties of our study by simulation. We report here in detail the results of this simulation study. Our main conclusion is that, for the particular set of families and markers used in our screen, an MLS of 3.2 carries a genome-wide significance of 5% (that is, there is a 5% probability of observing at least one false hit, above this threshold in a complete genome screen). This value is closer to the familiar limit of 3.0, originally suggested by Morton [1955; Am J Hum Genet 7:277-318] than to the more stringent limit of 4.0 recently proposed by Lander and Kruglyak [1995; Nat Genet 11:241-247]. This is somewhat reassuring, in view of the very large sample sizes that would be necessary to achieve adequate power to detect linkage at the more stringent threshold.

On the finite-sample distribution of modified portmanteau tests for radomness of a Gaussian time series

This paper proposes three modified portmanteau tests based on the applications of the Fisher (1921) and Hotelling (1953) transformations to sample autocorrelations. The test statistics are asymptotically chi-square. Our simulation results indicate that the empirical significance levels of the proposed tests are more reliable than the portmanteau tests of Box & Pierce (1970), Ljung & Box (1978) and Dufour & Roy (1986) when the sample size is small (15 ≤ n ≤ 40) and the number of sample autocorrelations is large (m ≥ 7).

Portmanteau tests of randomness and Jenkins' variance-stabilizing transformation

This paper proposes a new portmanteau test based on an application of the Jenkins' variance-stabilizing transformation to the sample autocorrelations. Our simulation results indicate that the empirical significance levels of the proposed test are more accurate than those of the Ljung-Box and the Dufour-Roy portmanteau tests.

Statistical methods for the beta-binomial model in teratology.

The beta-binomial model is widely used for analyzing teratological data involving littermates. Recent developments in statistical analyses of teratological data are briefly reviewed with emphasis on the model. For statistical inference of the parameters in the beta-binomial distribution, separation of the likelihood introduces an likelihood inference. This leads to reducing biases of estimators and also to improving accuracy of empirical significance levels of tests. Separate inference of the parameters can be conducted in a unified way.

Comparison of two modified portmanteau tests for model adequacy

The usual modified portmanteau test for model adequacy was proposed by Davies et al. [4] and Ljung and Box [13]. The portmanteau test is widely used for testing the adequacy of fitted ARIMA models and is incorporated directly into most computer packages which feature ARIMA modelling such as MINITAB, SAS, SPSS, BMDP and SCA. However this modification is not suitable for the multivariate case, so Li and McLeod [11] developed another modified portmanteau test statistic. This paper compares the two proposed modification in the univariate case. Our comparison suggests that the Li-McLeod modification is preferable since it yields a slightly conservative test whereas the Ljung-Box test errs in the opposite direction. Also, for moderate to large samples our results suggest that the empirical significance levels of the Li-McLeod test are closer to their nominal levels.