Standard Hypothesis Testing Research Articles

Understanding the Practical Advantages of Modern ANOVA Methods

Examined the fundamental problems associated with standard hypothesis testing techniques. This article explains why many articles have failed to detect problems due to nonnormality and discusses the basics of modern methods aimed at correcting these problems. Based on hundreds of published articles, it is now known that when groups differ, the analysis of variance (ANOVA) F test and related techniques that assume normality and homoscedasticity (equal variances) can perform poorly. In practical terms, if researchers interested in clinical child and adolescent psychology want to detect important differences among groups and accurately assess how the groups differ, and by how much, modern technology has much to offer. In fact, even highly nonsignificant results based on an F test can become significant. Moreover, modern methods offer improved control over the probability of a Type I error and more accurate confidence intervals.

On pooling data summaries in the absence of interactions “response-by-study”

The standard hypothesis testing procedure in meta-analysis (or multi-center clinical trials) in the absence of treatment-by-center interaction relies on approximating the null distribution of the standard test statistic by a standard normal distribution. For relatively small sample sizes, the standard procedure has been shown by various authors to have poor control of the type I error probability, leading to too many liberal decisions. In this article, two test procedures are proposed, which rely on thet—distribution as the reference distribution. A simulation study indicates that the proposed procedures attain significance levels closer to the nominal level compared with the standard procedure.

EFFECT-BASED INTERPRETATION OF TOXICITY TEST DATA USING PROBABILITY AND COMPARISON WITH ALTERNATIVE METHODS OF ANALYSIS

A methodology is described that incorporates the intra- and intertest variabilityand the biological effect of bioassay data in evaluating the toxicity of single and multiple tests for regulatory decision-making purposes. The single- and multiple-test regulatory decision probabilities were determined from t values (n − 1, one-tailed) derived from the estimated biological effect and the associated standard error at the critical sample concentration. Single-test regulatory decision probabilities below the selected minimum regulatory decision probability identify individual tests as noncompliant. A multiple-test regulatory decision probability is determined by combining the regulatory decision probability of a series of single tests. A multiple-test regulatory decision probability below the multiple-test regulatory decision minimum identifies groups of tests in which the magnitude and persistence of the toxicity is sufficient to be considered noncompliant or to require enforcement action. Regulatory decision probabilities derived from the t distribution were compared with results based on standard and bioequivalence hypothesis tests (t tests and no-observed-effect concentrations) using single- and multiple-concentration toxicity test data from an actual national pollutant discharge elimination system monitoring program. The analysis demonstrates that the use of regulatory decision probabilities effectively incorporated the precision of the effect estimate into regulatory decisions at a fixed level of effect. Also, probability-based interpretation of toxicity tests provides incentive to laboratories to produce, and permit holders to use, high-quality, precise data, particularly when multiple tests are used in regulatory decisions. These results are contrasted with standard and bioequivalence hypothesis tests in which the intratest precision is a determining factor in setting the biological effect used for regulatory decisions.

Effect‐based interpretation of toxicity test data using probability and comparison with alternative methods of analysis

AbstractA methodology is described that incorporates the intra‐ and intertest variabilityand the biological effect of bioassay data in evaluating the toxicity of single and multiple tests for regulatory decision‐making purposes. The single‐ and multiple‐test regulatory decision probabilities were determined from t values (n − 1, one‐tailed) derived from the estimated biological effect and the associated standard error at the critical sample concentration. Single‐test regulatory decision probabilities below the selected minimum regulatory decision probability identify individual tests as noncompliant. A multiple‐test regulatory decision probability is determined by combining the regulatory decision probability of a series of single tests. A multiple‐test regulatory decision probability below the multiple‐test regulatory decision minimum identifies groups of tests in which the magnitude and persistence of the toxicity is sufficient to be considered noncompliant or to require enforcement action. Regulatory decision probabilities derived from the t distribution were compared with results based on standard and bioequivalence hypothesis tests (t tests and no‐observed‐effect concentrations) using single‐ and multiple‐concentration toxicity test data from an actual national pollutant discharge elimination system monitoring program. The analysis demonstrates that the use of regulatory decision probabilities effectively incorporated the precision of the effect estimate into regulatory decisions at a fixed level of effect. Also, probability‐based interpretation of toxicity tests provides incentive to laboratories to produce, and permit holders to use, high‐quality, precise data, particularly when multiple tests are used in regulatory decisions. These results are contrasted with standard and bioequivalence hypothesis tests in which the intratest precision is a determining factor in setting the biological effect used for regulatory decisions.

A Three-step Method for Choosing the Number of Bootstrap Repetitions

This paper considers the problem of choosing the number of bootstrap repetitions B for bootstrap standard errors, confidence intervals, confidence regions, hypothesis tests, p-values, and bias correction. For each of these problems, the paper provides a three-step method for choosing B to achieve a desired level of accuracy. Accuracy is measured by the percentage deviation of the bootstrap standard error estimate, confidence interval length, test's critical value, test's p-value, or bias-corrected estimate based on B bootstrap simulations from the corresponding ideal bootstrap quantities for which B=∞. The results apply quite generally to parametric, semiparametric, and nonparametric models with independent and dependent data. The results apply to the standard nonparametric iid bootstrap, moving block bootstraps for time series data, parametric and semiparametric bootstraps, and bootstraps for regression models based on bootstrapping residuals. Monte Carlo simulations show that the proposed methods work very well.

Testing for statistical and market efficiency when forecast errors are non-normal: the NFL betting market revisited

This paper examines the efficiency of the National Football League (NFL) betting market. Previous econometric work on the subject has been based on least squares estimators, and has been marred by non-normal errors, which have cast doubt on standard hypothesis testing procedures. It is found that the negative binomial distribution provides a good description of points scored in NFL football games, but no significant departures from market efficiency were discovered.

Hypothesis testing of hazard ratio parameters in marginal models for multivariate failure time data.

Marginal hazard models for multivariate failure time data have been studied extensively in recent literature. However, standard hypothesis test statistics based on the likelihood method are not exactly appropriate for this kind of model. In this paper, extensions of the three commonly used likelihood hypothesis test statistics are discussed. Generalized Wald, generalized score and generalized likelihood ratio tests for hazard ratio parameters in a marginal hazard model for multivariate failure time data are proposed and their asymptotic distributions examined. The finite sample properties of these statistics are studied through simulations. The proposed method is applied to data from Busselton Population Health Surveys.

A modular methodology for fast fault detection and classification in power systems

This paper presents a modular yet integrated approach to the problem of fast fault detection and classification. Although the specific application example studied here is a power system, the method would be applicable to arbitrary dynamic systems. The approach is quite flexible in the sense that it can be model-based or model-free. In the model-free case, we emphasize the use of concepts from signal processing and wavelet theory to create fast and sensitive fault indicators. If a model is available then conventionally generated residuals can serve as fault indicators. The indicators can then be analyzed by standard statistical hypothesis testing or by artificial neural networks to create intelligent decision rules. After a detection, the fault indicator is processed by a Kohonen network to classify the fault. The approach described here is expected to be of wide applicability. Results of computer experiments with simulated faulty transmission lines are included.

A proposal for minimum detectable compartment in MIRD dosimetry modelling

The accuracy of radiation dose estimates from radiopharmaceutical administrations has recently become more important for three main reasons: (i) clinical providers are demanding more information on diagnostic procedures; (ii) regulatory groups are scrutinizing dosimetry for research subjects; and (iii) accurate organ doses are crucial in therapeutic administrations. These dose estimates are a sensitive function of the residence times. Because most clinical data acquisition protocols are limited to the first 24 h after dose administration, the area under the remainder of the time - activity curve (TAC) must be estimated. Estimation methods range from assuming physical decay only (overly conservative) to extrapolating end point physiological kinetics (overly liberal). This study demonstrates how much the results from these two methods vary and develops an alternative method which more accurately estimates this remainder term. A method, called the minimum detectable compartment (MDC), is constructed so that an accurate dose estimate can be made with a realistic measure of the remainder term. The method for determining MDC uses standard hypothesis testing. Using an analogue of the traditional minimal detectable activity calculation, a model with and without constant compartments is fitted to the TAC. The size of the constant compartment is varied until the relative likelihood of the two models meets the desired measure of power and sensitivity. Computer simulations of a simple mono-exponential are used to demonstrate the MDC as a function of the model, the number of data points, the range of the data and the noise in the data. The MDC is a very sensitive function of the data range. It falls by more than 50% when the data range is increased from two to three half-lives. In addition, the MDC is moderately sensitive to the noise in the data and relatively insensitive to the number of data points. These findings suggest that the MDC method can also be uses a priori to indicate what type of data collection regimen is necessary to achieve a certain accuracy.

The effect of Gaussian error in object recognition

In model based recognition, the goal is to locate an instance of one or more known objects in an image. The problem is compounded in real images by the presence of clutter, occlusion, and sensor error, which can lead to false negatives, failures to recognize the presence of the object, and false positives, in which the algorithm incorrectly identifies an occurrence of the object. The probability of either event is affected by parameters within the recognition algorithm, which are almost always chosen in an ad-hoc fashion. The effect of the parameter values on the likelihood that the recognition algorithm will make a mistake are usually not understood explicitly. To address the problem, we explicitly model the noise that occurs in the image. In a typical recognition algorithm, hypotheses about the position of the object are tested against the evidence in the image, and an overall score is assigned to each hypothesis. We use a statistical model to determine what score a correct or incorrect hypothesis is likely to have, and use standard binary hypothesis testing techniques to distinguish correct from incorrect hypotheses. Using this approach, we can compare algorithms and noise models, and automatically choose values for internal system thresholds to minimize the probability of making a mistake.

A test of purchasing power parity based on the largest principal component of real exchange rates of the main OECD economies

A unit root test is used to reject non-stationarity of the largest principal component of nine OECD bilateral real exchange rates over the recent float. Hence, support for purchasing power parity (PPP) over this period is obtained using a standard classical hypothesis test.

Exactly how manyp values is a picture worth? A commentary on Loftus’s plot-plus-error-bar approach

Loftus (1993b) has recently argued that hypothesis testing is largely irrelevant and should be replaced with a “plot-plus-error-bar” (PPE) presentation of the data. We agree that such figures can be useful. However, they do not preclude standard hypothesis testing procedures, and they should be used to supplement rather than to supplant the latter. The insufficiency of PPEs is most apparent in the case of mixed designs and factorial within-subject designs. In such cases, there is no single value that is appropriate to represent the standard error.

Unit root hypothesis, new classical and Keynesian models

Following Perron ( Econometrica, 1989, 57, 1361–1401) the variables in the new classical and Keynesian output equations are shown to be trend stationary. Therefore these models are evaluated for the United States with the standard non-nested hypothesis tests. The majority of these tests have rejected the new classical in favour of the Keynesian approach.

On rank statistics: an approach via metrics on the permutation group

Abstract This paper investigates a general construction that generates many rank test statistics, for most of the standard nonparametric hypothesis testing situations. The construction uses metrics on permutations in a new way: the induced rank test statistics is a distance between two sets of permutations. These sets of permutations have group-theoretic descriptions, and the resulting rank statistic metrizes a quotient space of the permutation group. This construction is applied systematically to the two-sample and multi-sample location problems, the two-way layout problem, the one-sample location problem, the two-sample dispersion problem with equal medians, and the problem of testing for trend. The construction gives rise to many familiar rank test statistics for these six hypothesis testing situations, and also produces some plausible new test statistics. Additionally, the method enables one to ‘extend’ rank statistics to other hypothesis testing situations, in the sense that it creates families of statistics, for the six given settings, that are linked by their use of the same metric. Some connections with standard nonparametric theory are discussed. In the course of implementing the general construction, certain combinatorial and group-theoretic properties of metrics on permutations are studied, that play key roles in the computation of the induced rank statistics.

Comparison of General Circulation Model and Observed Regional Climates: Daily and Seasonal Variability

Abstract Validation of general circulation model (GCM) current climate simulations is important for further GCM development and application to climate change studies. So far, studies that compare GCM output with observations have focused primarily on large-scale spatial averages of the surface climate variables. Here we discuss two approaches to compare output of individual GCM grid boxes with local station observations near the surface and in the free troposphere. The first approach, proposed by Chervin, involves the application of standard parametric statistical analysis and hypothesis testing procedures. The second approach is nonparametric in the sense that no ideal distributions are postulated a priori to ascertain significance of the difference of mean temperature or the ratio of the temperature variance between model grid boxes and local stations. Instead, station observations are first subjected to a bootstrap technique and then used to define a unique set of distributions and confidence limits fo...

Model building and judgment: Implications for benefit transfers with travel cost models

Benefit transfer is the process by which researchers take recreational demand models or other models which are estimated for one site or region and apply them to another site or region. Model estimation is frequently viewed as a simple exercise in classical statistics. Here it is argued that, while standard hypothesis testing plays a role in model estimation, it may be less important than the researcher's judgment about how the model ought to work. The power of economics lies principally with the logic of theory, and then with the strength of empirical evidence. These arguments are illustrated using the travel cost approach to estimating the demand for recreation. There are several major research issues (for example, the role of substitute prices) which influence the final estimates of benefits but which must be determined at least in part by the researcher. The implication for benefit transfer is that this is also within the realm of judgment for the researcher, but that it will be more acceptable when judgment is recognized. That is, the original benefit estimates are tailored for a specific application. They cannot be treated as if they came from a strictly random process. Recognizing this tailoring process will put the researcher in a better position for transferring model results.

Comparing Non-Nested Models for Contingency Tables

Many different models of contingency tables have been developed in recent years. Standard nested hypothesis tests are based on combining the alternatives in a comprehensive model. This is often difficult to accomplish, in which case standard hypothesis tests are unusable. This paper discusses an alternative approach, non-nested hypothesis tests, and extends the tests that have been developed for normal regression models to models for count data. Four models of the Glass mobility table are compared, including both loglinear and latent class models. The non-nested tests yield definite conclusions that are notpossible using conventional tests, suggesting that they are of practical value in the analysis of contingency tables and other count data.

Exact statistical tests for any carcinogenic effect in animal bioassays: II. Age-adjusted tests

Statistical methods are discussed for application in animal carcinogenesis bioassays that test a general null hypothesis that response frequencies are independent of the treatment level for all tumor endpoints, sexes, and species considered in the bioassay, conditional on survival. These methods are similar to those proposed earlier by Farrar and Crump (1988, Fundam. Appl. Toxicol. 11, 652–663) which involve test statistics that are functions of p values from multiple standard statistical hypothesis tests, and evaluation of significance using randomization. Refinements include the introduction of an exact incidental tumor test analogous to the asymptotic test of Hoel and Walburg (1972, J. Natl. Cancer Inst. 49, 361–372), which takes into account the possibility that any differences among treatment groups in response rates result from differences in survival that is independent of tumors. Additional consideration is given to selection of test statistics, and a test is proposed that is sensitive specifically to cases where the same tumor endpoints are affected in multiple sexes and species. Applications of the procedures to results of National Toxicology Program (NTP) bioassays concerning decabromodiphenyl oxide and iodinated glycerol illustrate the sensitivity of tests based on different test statistics to distinct alternative hypotheses. When compared with the qualitative conclusions of the NTP regarding the strength of evidence for a carcinogenic effect in the same bioassays, the results presented suggest that the methods can be useful in clarifying otherwise equivocal evidence for carcinogenicity. Various difficulties are discussed relevant to the construction of exact combined fatal and incidental tumor tests analogous to the tests of Peto et al. (1980 , IARC Monogr. Suppl. 2).

A simple statistical method for the comparison of accumulated dose in ESR dating

It is generally recognized that estimates of accumulated dose (AD) for ESR dating are vulnerable to variations in sample preparation and measurement procedures: it is therefore desirable to be able to compare AD's derived by different methods as a first step in establishing universally acceptable methods. However, because AD is calculated by dividing two variables, the distribution of AD is non-linear and standard linear hypothesis tests cannot be applied directly. The method developed here by the addition of another assumption (which can be tested) allows AD's to be compared using Student's t -test. Hence, sample preparation and measurement procedures can be tested for equivalence. The technique also allows AD of different samples to be compared, provided the values of the additional radiation are the same or similar for each sample. Confidence limits for AD may also be derived. Existing linear regression programs can be easily modified to accomodate the method.

Statistical methods for model discrimination. Applications to gating kinetics and permeation of the acetylcholine receptor channel

Methods are described for discrimination of models of the gating kinetics and permeation of single ionic channels. Both maximum likelihood and regression procedures are discussed. In simple situations, where models are nested, standard hypothesis tests can be used. More commonly, however, non-nested models are of interest, and several procedures are described for model discrimination in these cases, including Monte Carlo methods, which allow the comparison of models at significance levels of choice. As an illustration, the methods are applied to single-channel data from acetylcholine receptor channels.