Uniformly Most Powerful Unbiased Test Research Articles

Finite-Sample Two-Group Composite Hypothesis Testing via Machine Learning

In the problem of composite hypothesis testing, identifying the potential uniformly most powerful (UMP) unbiased test is of great interest. Beyond typical hypothesis settings with exponential family, it is usually challenging to prove the existence and further construct such UMP unbiased tests with finite sample size. For example in the COVID-19 pandemic with limited previous assumptions on the treatment for investigation and the standard of care, adaptive clinical trials are appealing due to ethical considerations, and the ability to accommodate uncertainty while conducting the trial. Although several methods have been proposed to control Type I error rates, how to find a more powerful hypothesis testing strategy is still an open question. Motivated by this problem, we propose an automatic framework of constructing test statistics and corresponding critical values via machine learning methods to enhance power in a finite sample. In this article, we particularly illustrate the performance using Deep Neural Networks (DNN) and discuss its advantages. Simulations and two case studies of adaptive designs demonstrate that our method is automatic, general and prespecified to construct statistics with satisfactory power in finite-sample. Supplemental materials are available online including R code and an R shiny app.

Separating Function Estimation Tests: A New Perspective on Binary Composite Hypothesis Testing

In this paper, we study some relationships between the detection and estimation theories for a binary composite hypothesis test <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> against <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> and a related estimation problem. We start with a one-dimensional (1D) space for the unknown parameter space and one-sided hypothesis problems and then extend out results into more general cases. For one-sided tests, we show that the uniformly most powerful (UMP) test is achieved by comparing the minimum variance and unbiased estimator (MVUE) of the unknown parameter with a threshold. Thus for the case where the UMP test does not exist, the MVUE of the unknown parameter does not exist either. Therefore for such cases, a good estimator of the unknown parameter is deemed as a good decision statistic for the test. For a more general class of composite testing with multiple unknown parameters, we prove that the MVUE of a separating function (SF) can serve as the optimal decision statistic for the UMP unbiased test where the SF is continuous, differentiable, positive for all parameters under <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> and is negative for the parameters under <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> . We then prove that the UMP unbiased statistic is equal to the MVUE of an SF. In many problems with multiple unknown parameters, the UMP test does not exist. For such cases, we show that if one detector between two detectors has a better receiver operating characteristic (ROC) curve, then using its decision statistic we can estimate the SF more ε-accurately, in probability. For example, the SF is the signal-to-noise ratio (SNR) in some problems. These results motivate us to introduce new suboptimal SF-estimator tests (SFETs) which are easy to derive for many problems. Finally, we provide some practical examples to study the relationship between the decision statistic of a test and the estimator of its corresponding SF.

Optimal significance analysis of microarray data in a class of tests whose null statistic can be constructed

Microarray data often consist of a large number of genes and a small number of replicates. We have examined testing the null hypothesis of equality of mean for detecting differentially expressed genes. The p-value for each gene is often estimated using permutation samples not only for the target gene but also for other genes. This method has been widely used and discussed. However, direct use of the permutation method for the p-value estimation may not work well, because two types of genes are mixed in the sample; some genes are differentially expressed, whereas others are not. To overcome this difficulty, various methods for appropriately generating null permutation samples have been proposed. In this paper, we consider two classes of test statistics that are naturally modified to null statistics. We then obtain the uniformly most powerful (UMP) unbiased tests among these classes. If the underlying distribution is symmetric, the UMP unbiased test statistic is similar to that proposed by Pan (Bioinformatics 19:1333–1340, 2003). Under another condition, the UMP unbiased test statistic has a different formula with one more degree of freedom and therefore is expected to give a more powerful test and a more accurate p-value estimation from a modified null statistic. In microarray data, because the number of replicates is often small, differences in the degree of freedom will produce large effects on the power of test and the variance of the p-value estimation. Some simulation studies and real data analyses are illustrated to investigate the performances of the methods.

Testing the Pareto against the lognormal distributions with the uniformly most powerful unbiased test applied to the distribution of cities

Fat-tail distributions of sizes abound in natural, physical, economic, and social systems. The lognormal and the power laws have historically competed for recognition with sometimes closely related generating processes and hard-to-distinguish tail properties. This state-of-affair is illustrated with the debate between Eeckhout [Amer. Econ. Rev. 94, 1429 (2004)] and Levy [Amer. Econ. Rev. 99, 1672 (2009)] on the validity of Zipf's law for US city sizes. By using a uniformly most powerful unbiased (UMPU) test between the lognormal and the power-laws, we show that conclusive results can be achieved to end this debate. We advocate the UMPU test as a systematic tool to address similar controversies in the literature of many disciplines involving power laws, scaling, "fat" or "heavy" tails. In order to demonstrate that our procedure works for data sets other than the US city size distribution, we also briefly present the results obtained for the power-law tail of the distribution of personal identity (ID) losses, which constitute one of the major emergent risks at the interface between cyberspace and reality.

IRT-Based Measurement of Short-Term Changes of Ability, With an Application to Assessing the “Mozart Effect”

The article extends and applies previous approaches by Klauer and Fischer to the statistical evaluation of ability changes in tests conforming to the Rasch model (RM). Exact uniformly most powerful unbiased (UMPU) hypothesis tests and uniformly most accurate (UMA) confidence intervals (CIs) for the amount of change can be constructed for each testee separately. Under the H0 of no change, the individual p values of testees can be aggregated also within groups. The current results extend this method by an estimator for a group effect, an exact UMPU test of its significance, a corresponding UMA CI, and a likelihood ratio test for group comparisons. All these methods are applied to two sets of data concerning a highly controversial issue in literature: the so-called Mozart Effect (ME). As test material, a unidimensional test of spatial ability by Gittler and Arendasy was presented to two samples of testees (Studies I and II). On the basis of these data, the occurrence of a short-term effect of the experimental condition can be demonstrated which, however, seems to be related to the relaxation between pretest and posttest rather than to the musical priming.

UMPU tests based on sequential order statistics

Sequential order statistics with conditional proportional hazard rates form a regular exponential family in the model parameters. This finding is used to establish uniformly most powerful unbiased (UMPU) tests for a variety of hypotheses.

Modified p-Value of Two-Sided Test for Normal Distribution with Restricted Parameter Space

This article proposes a modified p-value for the two-sided test of the location of the normal distribution when the parameter space is restricted. A commonly used test for the two-sided test of the normal distribution is the uniformly most powerful unbiased (UMPU) test, which is also the likelihood ratio test. The p-value of the test is used as evidence against the null hypothesis. Note that the usual p-value does not depend on the parameter space but only on the observation and the assumption of the null hypothesis. When the parameter space is known to be restricted, the usual p-value cannot sufficiently utilize this information to make a more accurate decision. In this paper, a modified p-value (also called the rp-value) dependent on the parameter space is proposed, and the test derived from the modified p-value is also shown to be the UMPU test.

Bayesian approach to estimation of accuracy in two-sided testing

For a two-sided hypothesis testing, it is known that there does not exist UMP tests. Uniformly most powerful unbiased (UMPU) tests and likelihood ratio test are the common approaches to two-sided testing. The p-values of these tests are usually used as evidence against null hypothesis. However, there are criticisms of p-values as a measure of evidence against null hypothesis for the two-sided testing problem in literature. Thus, in this paper, evidence measures derived from Bayesian approach are proposed to be the replacements of p-values and are investigated from both decision and testing perspectives. From decision theoretic framework, the proposed evidence measures can be demonstrated as admissible estimators, however, p-value of UMPU test are not admissible estimators; from testing aspect, the tests derived from p-values and the proposed evidence measures are shown to be UMPU tests. The Bayes estimator is better than p-value from theoretical aspect and it has the same merit as the p-value in testing point of view. Therefore, evidence measures derived from Bayesian approach are recommended in this paper.

Fuzzy and Randomized Confidence Intervals and P-Values

The optimal hypothesis tests for the binomial distribution and some other discrete distributions are uniformly most powerful (UMP) one-tailed and UMP unbiased (UMPU) two-tailed randomized tests. Conventional confidence intervals are not dual to randomized tests and perform badly on discrete data at small and moderate sample sizes. We introduce a new confidence interval notion, called fuzzy confidence intervals, that is dual to and inherits the exactness and optimality of UMP and UMPU tests. We also introduce a new P-value notion, called fuzzy P-values or abstract randomized P-values, that also inherits the same exactness and optimality.

A UMPU TEST FOR COMPARING MEANS OF TWO NEGATIVE BINOMIAL DISTRIBUTIONS

A common problem in the biological sciences is to compare the means of two populations of count data. This paper introduces an uniformly most powerful (UMP) unbiased test for testing the equality of means for independent samples of count data from two negative binomial populations in the presence of a common dispersion parameter. The size and power of the UMP unbiased test is compared to several other known tests. The proposed methodology is illustrated with an application to real data.

Power and sample size requirements for tests of differences between two Poisson rates

The statistic (y−x)/(y + x)1/2 is often used as an approximate test for comparing two Poisson observations. When x and y come from different sized sampling frames, a uniformly most powerful unbiased (UMPU) test is obtained using the binomial distribution of the conditional distribution of y given x + y. Shiue and Bain showed that a test based on the normal approximation of the binomial distribution is nearly as powerful as the UMPU test and presented power and sample size formulae. Here we present an alternative test and show that, when the Poisson rate is greater for the larger sampling frame, it has higher power than the Shiue and Bain test. Power and sample size formulae are given.

Generalized Likelihood Ratio Tests and Uniformly Most Powerful Tests

Abstract For testing a one-sided hypothesis in a one-parameter family of distributions, it is shown that the generalized likelihood ratio (GLR) test coincides with the uniformly most powerful (UMP) test, assuming certain monotonicity properties for the likelihood function. In particular, the equivalence of GLR tests and UMP tests holds for one-parameter exponential families. In addition, the relationship between GLR and UMPU (UMP unbiased) tests is considered when testing two-sided hypotheses.

Are Uniformly Most Powerful Unbiased Tests Really Best?

Abstract In teaching the development of uniformly most powerful unbiased (UMPU) tests, one rarely discusses the performance of alternative biased tests. It is shown, through the comparison of two independent Bernoulli proportions, that a biased test (the Z test) can be more powerful than the UMPU test (Fisher's exact test—randomized) in a large region of the alternative parameter space. A more general example is also given.

The Relation between Score Tests and Approximate UMPU Tests in Exponential Models Common in Biometry

In several recent papers, both published and unpublished, the theory of score tests (Rao, 1973, pp. 415-420), or the equivalent C(a) test (Neyman, 1959), has been used to derive tests involving likelihoods based on exponential families which yield sufficient statistics. A recently published example is a paper by Day and Byar (1979) in which the test for combined 2 x 2 tables due to Cochran (1954) [sometimes called the Mantel-Haenszel (1959) statistic] is shown to be a 'logit score test'. Day and Byar also considered a second multiparameter test for a more general logistic model. We shall show that, in such cases, score tests can ordinarily be written down directly upon examination of the structure of the likelihood; neither differentiation of the likelihood nor matrix inversion is usually needed. Furthermore, for the case in which a single parameter is tested in the presence of nuisance parameters, the score test is simply the normal approximation to the uniformly most powerful unbiased (UMPU) test. Obviously, for one-parameter exponential families the score test is not only asymptotically locally most powerful but is also an approximation to the UMPU test (Efron, 1975).

An Exponential Subfamily which Admits UMPU Tests Based on a Single Test Statistic

Let $f(x: \theta) = a(x) \exp\{\theta_1u_1(x) + \theta_2u_2(x) + c(\theta)\}, \theta = (\theta_1, \theta_2) \in \ominus \subset R^2$, be a density with respect to the Lebesgue measure on the real line which characterizes a two-parameter exponential family of distributions. Let $(\theta_1, \eta_2)$ be the mixed parameters, where $\eta_2 = E\{u_2(X)\}$. Assume that $\theta_2$ can be represented as $\theta_2 = -\theta_1 \varphi'(\eta_2)$ where $\varphi'(\eta_2) = d\varphi(\eta_2)/d\eta_2$ for some function $\varphi(\eta_2)$. Let $(X_1, \cdots, X_n)$ be independent random variables having a common density $f(x: \theta)$ and set $T_i = \sum^n_{j=1} u_i(X_j), i = 1, 2$. It is shown that if $u_2(x)$ is a 1-1 function then the random variables $T_2$ and $Z_n = T_1 - n\varphi(T_2/n)$ are independent and that the statistic $Z_n$ is ancillary for $\theta_2$ in the presence of $\theta_1$ (i.e. the density of $Z_n$ depends on $\theta_1$ only). Furthermore, the density of $Z_n$ belongs to the one-parameter exponential family with natural parameter $\theta_1$. These results enable us to construct uniformly most powerful unbiased (UMPU) tests for various hypotheses concerning the parameter $\theta_1$ which are based on the statistic $Z_n$.

Methods and criteria for deciding whether specific-locus mutation-rate data in mice indicate a positive, negative, or inconclusive result

The binomial approximation of the UMPU (uniformly most powerful unbiased) test for the equality of 2 binomial proportions is shown to be a highly accurate and easily applied method for testing the hypothesis that a given mouse specific-locus mutation frequency is not higher than the spontaneous mutation frequency (43 mutations in 801 406 offspring, for males). Critical sample sizes have been calculated that show at a glance whether P < 0.05. The first hypothesis that the mutation frequency (induced + spontaneous) of treated mice is not higher than the spontaneous mutation frequency is combined with the second hypothesis that the induced mutation frequency of treated mice is no less than 4 times the historical-control mutation frequency to produce a multiple decision procedure with 4 possible decisions: inconclusive result, negative result, positive result, and weak mutagen. Critical sample sizes for the second hypothesis, also with P < 0.05, are combined with those for the first hypothesis into a grid that permits rapid evaluation of data according to these criterea. The justification for using these criteria in reaching decisions, assuming a high level of exposure has been given, is the practical necessity of rapidly determining which chemicals are potent mutagens. Positive results can become apparent in relatively small samples. Larger samples, of at least 11 166 offspring, are required to obtain a negative result. If samples of 18 000 are routinely collected (unless positive results are found earlier), 75% of tests of chemicals that are non-mutagens will give a negative result. If the question being asked is not whether a chemical induces gene mutations but, rather, whether the exposure received by humans causes any important risk from gene mutations, a much smaller sample size may be acceptable, under certain conditions. A comparison of the relative efficiencies of the specific-locus test (for gene mutations and small deficiencies) and the heritable-translocation test (for transmissible chromosome rearrangements), in detecting the same proportional increases over the spontaneous frequencies of their respective types of genetic damage, shows that less work is involved in reaching a conclusive result in the specific-locus test. Proposed specific-locus tests using biochemical markers are at a considerable statistical disadvantage compared with the standard test (using 7 visible markers) for which there is available a very large historical control showing a very low mutation rate.

Sonar Array Detection of Gaussian Signals in Gaussian Noise of Unknown Power

A statistical test is postulated for detecting, with an M-element hydrophone array, a Gaussian signal in spatially independent Gaussian noise of unknown power. The test is an extension of the uniformly-most-powerful (UMP) unbiased test for a two-element array. The output signal-to-noise ratio of the test is calculated and, for a large number of independent space-time samples, is shown to be no better than a mean-level detector (MLD). Receiver operating characteristic curves (ROC) for the MLD are computed and compared to the ROC curves for the optimum (Bayes) parametric detector. The input signal-to-noise power ratios required to provide a detection probability of 0.5 differ by less than 0.2 dB for a fifty-element array with wide variation in false-alarm probability and time-bandwidth product. This result suggests that both the extended bivariate UMP unbiased test and the MLD perform close to the unknown UMP unbiased test for independence of a multivariate Gaussian distribution.

Locally Unbiased Type M Test

Summary In certain tests of hypotheses concerning more than one parameter, it is found that the criterion of unbiasedness cannot be satisfied. In such situations, it is proposed to make the test locally unbiased in a modified sense, termed Type M. Here the power surface in the parametric space does not always lie above the level of significance of the test. However, the mean value of the power, in a small neighbourhood of the null point, exceeds the significance level. This paper demonstrates that the uniformly most powerful unbiased (UMPU) test (based on the sample sum of squares) for a normal variance (Neyman and Pearson, 1936; Rao, 1952; Ramachandran, 1958) is Type M unbiased against the wider class of alternatives permitting the means of the observations to differ, and similarly that the UMPU test (based on the sample variance ratio) for a population variance ratio is Type M unbiased against the same widened class of alternatives.