Entropy-based Test Research Articles

모수적 엔트로피 추정량과 비모수적 엔트로피 추정량에 기초한 정규분포에 대한 적합도 검정

본 논문에서는 모수적과 비모수적 엔트로피 추정량들에 기초한 정규분포에 대한 적합도 검정을 다룬다. 정규분포의 엔트로피에 대한 모수적 추정량으로 사용할 최소분산비편향추정량을 유도한다. 이 추정량과 대립가설 하에서의 자료생성분포에 대한 비모수적 엔트로피 추정량으로 표본엔트로피와 이것의 변형된 추정량들을 이용하여 검정통계량들을 구축했고 이 검정통계량들을 사용하는 새로운 엔트로피 기반 적합도 검정들을 제시한다. 제안한 검정들의 기각값들을 모의실험을 통해 추정해서 표의 형태로 제시한다. 성능의 조사를 위해 수행한 모의실험에서 제안한 검정들이 기존의 Vasicek (1976) 검정보다는 더 좋은 검정력을 가지는 것으로 나타난다. 응용에서 새로운 검정들이 정규성 검정을 위한 경쟁적인 도구로 시용될 수 있을 것으로 기대된다. In this paper, we deal with testing goodness-of-fit for normal distribution based on parametric and nonparametric entropy estimators. The minimum variance unbiased estimator for the entropy of the normal distribution is derived as a parametric entropy estimator to be used for the construction of a test statistic. For a nonparametric entropy estimator of a data-generating distribution under the alternative hypothesis sample entropy and its modifications are used. The critical values of the proposed tests are estimated by Monte Carlo simulations and presented in a tabular form. The performance of the proposed tests under some selected alternatives are investigated by means of simulations. The results report that the proposed tests have better power than the previous entropy-based test by Vasicek (1976). In applications, the new tests are expected to be used as a competitive tool for testing normality.

Power Investigation of the Entropy-Based Test of Fit for Inverse Gaussian Distribution by the Information Discrimination Index

Inverse Gaussian distribution is widely used in applications to analyze and model right-skewed data. To assess the appropriateness of the distribution prior to data analysis, Mudholkar and Tian (2002) proposed an entropy-based test of fit. The test is based on the entropy power fraction(EPF) index suggested by Gokhale (1983). The simulation results report that the power of the entropy-based test is superior compared to other goodness-of-fit tests; however, this observation is based on the small-scale simulation results on the standard exponential, Weibull W(1;2) and lognormal LN(0:5;1) distributions. A large-scale simulation should be performed against various alternative distributions to evaluate the power of the entropy-based test; however, the use of a theoretical method is more effective to investigate the powers. In this paper, utilizing the information discrimination(ID) index defined by Ehsan et al. (1995) as a mathematical tool, we scrutinize the power of the entropy-based test. The selected alternative distributions are the gamma, Weibull and lognormal distributions, which are widely used in data analysis as an alternative to inverse Gaussian distribution. The study results are provided and an illustrative example is analyzed.

Testing normality based on new entropy estimators

In this paper, we first introduce two new estimators for estimating the entropy of absolutely continuous random variables. We then compare the introduced estimators with the existing entropy estimators, including the first of such estimators proposed by Dimitriev and Tarasenko [On the estimation functions of the probability density and its derivatives, Theory Probab. Appl. 18 (1973), pp. 628–633]. We next propose goodness-of-fit tests for normality based on the introduced entropy estimators and compare their powers with the powers of other entropy-based tests for normality. Our simulation results show that the introduced estimators perform well in estimating entropy and testing normality.

On the Choice of Nonparametric Entropy Estimator in Entropy-Based Goodness-of-Fit Test Statistics

Entropy-based goodness-of-fit test statistics can be established by estimating the entropy difference or Kullback–Leibler information, and several entropy-based test statistics based on various entropy estimators have been proposed. In this article, we first give comments on some problems resulting from not satisfying the moment constraints. We then study the choice of the entropy estimator by noting the reason why a test based on a better entropy estimator does not necessarily provide better powers.

역가우스분포에 대한 쿨백-라이블러 정보 기반 적합도 검정

본 논문에서는 위치와 척도모수가 모두 알려지지 않은 역가우스분포에 대한 적합도 검정으로 기존에 개발된 엔트로피 기반 검정을 확장한 쿨백-라이블러 정보 기반 적합도 검정을 소개한다. 역가우스분포에 대한 단순 또는 복합 영가설을 검정하기 위한 4가지 형태의 검정통계량을 제시하고 검정통계량의 계산에 사용할 표본크기에 따른 윈도크기와 기각값을 모의실험을 통해 결정하여 표의 형태로 제공한다. 검정력 분석을 위해 수행한 모의실험의 결과에서 위치와 척도모수가 모두 알려진 역가우스분포에 대한 쿨백-라이블러 정보 기반 적합도 검정은 모든 대립분포와 표본크기에서 EDF 검정들보다 좋은 검정력을 가지는 것으로 나타난다. 위치모수 또는 척도모수만 알려진 역가우스분포에 대한 쿨백-라이블러 정보 기반 적합도 검정은 모든 대립분포에 대해서 표본크기가 커짐에 따라 검정력이 증가하는 경향을 보인다. 위치와 척도모수가 모두 알려지지 않은 역가우스분포에 대한 쿨백-라이블러 정보 기반 적합도 검정은 대체적으로 엔트로피 기반 검정과 비슷한 수준의 검정력을 보이는 것으로 나타나고 이 결과를 통해서 두 검정은 동일함을 확인할 수 있다. The entropy-based test of fit for the inverse Gaussian distribution presented by Mudholkar and Tian(2002) can only be applied to the composite hypothesis that a sample is drawn from an inverse Gaussian distribution with both the location and scale parameters unknown. In application, however, a researcher may want a test of fit either for an inverse Gaussian distribution with one parameter known or for an inverse Gaussian distribution with both the two partameters known. In this paper, we introduce tests of fit for the inverse Gaussian distribution based on the Kullback-Leibler information as an extension of the entropy-based test. A window size should be chosen to implement the proposed tests. By means of Monte Carlo simulations, window sizes are determined for a wide range of sample sizes and the corresponding critical values of the test statistics are estimated. The results of power analysis for various alternatives report that the Kullback-Leibler information-based goodness-of-fit tests have good power.

Entropy-based information gain approaches to detect and to characterize gene-gene and gene-environment interactions/correlations of complex diseases.

For complex diseases, the relationship between genotypes, environment factors, and phenotype is usually complex and nonlinear. Our understanding of the genetic architecture of diseases has considerably increased over the last years. However, both conceptually and methodologically, detecting gene-gene and gene-environment interactions remains a challenge, despite the existence of a number of efficient methods. One method that offers great promises but has not yet been widely applied to genomic data is the entropy-based approach of information theory. In this article, we first develop entropy-based test statistics to identify two-way and higher order gene-gene and gene-environment interactions. We then apply these methods to a bladder cancer data set and thereby test their power and identify strengths and weaknesses. For two-way interactions, we propose an information gain (IG) approach based on mutual information. For three-ways and higher order interactions, an interaction IG approach is used. In both cases, we develop one-dimensional test statistics to analyze sparse data. Compared to the naive chi-square test, the test statistics we develop have similar or higher power and is robust. Applying it to the bladder cancer data set allowed to investigate the complex interactions between DNA repair gene single nucleotide polymorphisms, smoking status, and bladder cancer susceptibility. Although not yet widely applied, entropy-based approaches appear as a useful tool for detecting gene-gene and gene-environment interactions. The test statistics we develop add to a growing body methodologies that will gradually shed light on the complex architecture of common diseases.

역가우스분포에 대한 변형된 엔트로피 기반 적합도 검정

이 논문에서는 역가우스분포의 적합을 위한 변형된 엔트로피 기반 검정을 제시한다. 이 검정은 자료생성분포와 역가우스분포의 엔트로피 차이에 기초를 두고 있으며 검정통계량은 엔트로피 차이의 추정량을 사용한다. 엔트로피 차이의 추정량은 자료생성분포에 대한 엔트로피 추정량으로 Vasicek의 표본엔트로피와 역가우스분포에 대한 엔트로피 추정량로 균일최소분산불편추정량을 사용하여 얻는다. 모의실험을 통해 얻은 표본크기와 윈도크기에 따른 검정통계량의 기각값들을 표의 형태로 제공한다. 제안한 검정의 검정력 알아보기 위해 여러 대립분포와 표본크기에 대해서 모의실험을 수행하고 기존의 엔트로피 기반 검정과 비교한다. This paper presents a modified entropy-based test of fit for the inverse Gaussian distribution. The test is based on the entropy difference of the unknown data-generating distribution and the inverse Gaussian distribution. The entropy difference estimator used as the test statistic is obtained by employing Vasicek's sample entropy as an entropy estimator for the data-generating distribution and the uniformly minimum variance unbiased estimator as an entropy estimator for the inverse Gaussian distribution. The critical values of the test statistic empirically determined are provided in a tabular form. Monte Carlo simulations are performed to compare the proposed test with the previous entropy-based test in terms of power.

Entropy Based Genetic Association Tests and Gene-Gene Interaction Tests

In the past few years, several entropy-based tests have been proposed for testing either single SNP association or gene-gene interaction. These tests are mainly based on Shannon entropy and have higher statistical power when compared to standard χ2 tests. In this paper, we extend some of these tests using a more generalized entropy definition, Rényi entropy, where Shannon entropy is a special case of order 1. The order λ (>0) of Rényi entropy weights the events (genotype/haplotype) according to their probabilities (frequencies). Higher λ places more emphasis on higher probability events while smaller λ (close to 0) tends to assign weights more equally. Thus, by properly choosing the λ, one can potentially increase the power of the tests or the p-value level of significance. We conducted simulation as well as real data analyses to assess the impact of the order λ and the performance of these generalized tests. The results showed that for dominant model the order 2 test was more powerful and for multiplicative model the order 1 or 2 had similar power. The analyses indicate that the choice of λ depends on the underlying genetic model and Shannon entropy is not necessarily the most powerful entropy measure for constructing genetic association or interaction tests.

Liquid: A detection-resistant covert timing channel based on IPD shaping

Covert timing channels provide a way to surreptitiously leak information from an entity in a higher-security level to an entity in a lower level. The difficulty of detecting or eliminating such channels makes them a desirable choice for adversaries that value stealth over throughput. When one considers the possibility of such channels transmitting information across network boundaries, the threat becomes even more acute. A promising technique for detecting covert timing channels focuses on using entropy-based tests. This method is able to reliably detect known covert timing channels by using a combination of entropy and conditional entropy to detect anomalies in shape and regularity, respectively. This dual approach is intended to make entropy-based detection robust against both current and future channels. In this work, we show that entropy-based detection can be defeated by a channel that intelligently and adaptively manipulates the metrics used for detection. Specifically, we propose a new passive covert channel that uses a portion of the inter-packet delays in a compromised stream to smooth out the shape distortions detected by the entropy test. As a passive channel, it is not as prone to regularity-based detection as previously proposed active channels. We introduce a model for analyzing the effect of our techniques on the entropy of the channel and empirically investigate the accuracy of the model. In network experiments and simulation, we validate this model and demonstrate that the proposed channel successfully evades entropy-based detection and other known tests while maintaining reasonable throughput.

Density-Based Empirical Likelihood Ratio Change Point Detection Policies

Nonparametric detection of a change in distribution is studied when observations are independent. We provide a general method for constructing distribution-free change point detection schemes that have approximately likelihood structures. This method utilizes principles of the maximum empirical likelihood (EL) approach to approximate powerful parametric likelihood ratios. The product of the approximation can be associated with entropy-based test statistics. Entropy-based tests have been well addressed in the literature in the context of powerful decision rules for goodness of fit. We extend the entropy-based technique, using the EL principles. CUSUM and Shiryayev–Roberts (SR) detection policies are shown to be powerful parametric likelihood tests for detecting a change in distribution. We apply the proposed method to obtain nonparametric forms of the CUSUM and SR procedures. A Monte Carlo study demonstrates that the proposed method provides very efficient tests when comparing with the classical procedures. An example based on a real data is provided to demonstrate implementation and effectiveness of the new tests.

A two-sample empirical likelihood ratio test based on samples entropy

Powerful entropy-based tests for normality, uniformity and exponentiality have been well addressed in the statistical literature. The density-based empirical likelihood approach improves the performance of these tests for goodness-of-fit, forming them into approximate likelihood ratios. This method is extended to develop two-sample empirical likelihood approximations to optimal parametric likelihood ratios, resulting in an efficient test based on samples entropy. The proposed and examined distribution-free two-sample test is shown to be very competitive with well-known nonparametric tests. For example, the new test has high and stable power detecting a nonconstant shift in the two-sample problem, when Wilcoxon's test may break down completely. This is partly due to the inherent structure developed within Neyman-Pearson type lemmas. The outputs of an extensive Monte Carlo analysis and real data example support our theoretical results. The Monte Carlo simulation study indicates that the proposed test compares favorably with the standard procedures, for a wide range of null and alternative distributions.

An entropy test for single-locus genetic association analysis

BackgroundThe etiology of complex diseases is due to the combination of genetic and environmental factors, usually many of them, and each with a small effect. The identification of these small-effect contributing factors is still a demanding task. Clearly, there is a need for more powerful tests of genetic association, and especially for the identification of rare effectsResultsWe introduce a new genetic association test based on symbolic dynamics and symbolic entropy. Using a freely available software, we have applied this entropy test, and a conventional test, to simulated and real datasets, to illustrate the method and estimate type I error and power. We have also compared this new entropy test to the Fisher exact test for assessment of association with low-frequency SNPs. The entropy test is generally more powerful than the conventional test, and can be significantly more powerful when the genotypic test is applied to low allele-frequency markers. We have also shown that both the Fisher and Entropy methods are optimal to test for association with low-frequency SNPs (MAF around 1-5%), and both are conservative for very rare SNPs (MAF<1%)ConclusionsWe have developed a new, simple, consistent and powerful test to detect genetic association of biallelic/SNP markers in case-control data, by using symbolic dynamics and symbolic entropy as a measure of gene dependence. We also provide a standard asymptotic distribution of this test statistic. Given that the test is based on entropy measures, it avoids smoothed nonparametric estimation. The entropy test is generally as good or even more powerful than the conventional and Fisher tests. Furthermore, the entropy test is more computationally efficient than the Fisher's Exact test, especially for large number of markers. Therefore, this entropy-based test has the advantage of being optimal for most SNPs, regardless of their allele frequency (Minor Allele Frequency (MAF) between 1-50%). This property is quite beneficial, since many researchers tend to discard low allele-frequency SNPs from their analysis. Now they can apply the same statistical test of association to all SNPs in a single analysis., which can be especially helpful to detect rare effects.

Nonparametric Entropy-Based Tests of Independence Between Stochastic Processes

This article develops nonparametric tests of independence between two stochastic processes satisfying β-mixing conditions. The testing strategy boils down to gauging the closeness between the joint and the product of the marginal stationary densities. For that purpose, we take advantage of a generalized entropic measure so as to build a whole family of nonparametric tests of independence. We derive asymptotic normality and local power using the functional delta method for kernels. As a corollary, we also develop a class of entropy-based tests for serial independence. The latter are nuisance parameter free, and hence also qualify for dynamic misspecification analyses. We then investigate the finite-sample properties of our serial independence tests through Monte Carlo simulations. They perform quite well, entailing more power against some nonlinear AR alternatives than two popular nonparametric serial-independence tests.

Empirical likelihood ratios applied to goodness-of-fit tests based on sample entropy

The likelihood approach based on the empirical distribution functions is a well-accepted statistical tool for testing. However, the proof schemes of the Neyman–Pearson type lemmas induce consideration of density-based likelihood ratios to obtain powerful test statistics. In this article, we introduce the distribution-free density-based likelihood technique, applied to test for goodness-of-fit. We focus on tests for normality and uniformity, which are common tasks in applied studies. The well-known goodness-of-fit tests based on sample entropy are shown to be a product of the proposed empirical likelihood (EL) methodology. Although the efficiency of test statistics based on classes of entropy estimators has been widely addressed in the statistical literature, estimation of the sample entropy has been not invariantly defined, and hence this estimation produces tests that are difficult to be applied to real data studies. The proposed EL approach defines clear forms of the entropy-based tests. Monte Carlo simulation results confirm the preference of the proposed method from a power perspective. Real data examples study the proposed approach in practice.

Can the electricity market be characterised by asymmetric behaviour?

In this paper, we test for asymmetric behaviour of industrial and residential electricity demand for the G7 countries, using the entropy-based test for symmetry suggested by [Racine, J., and Maasoumi, E., 2007. A versatile and robust metric entropy test of time-reversibility, and other hypotheses. Journal of Econometrics 138(2), 547–567; Racine, J., and Maasoumi, E., 2008. A robust entropy-based test of asymmetry for discrete and continuous processes. Econometric Reviews 28, 246–261], the Triples test of [Randles, R., Flinger, M., Policello, G., and Wolfe, D., 1980. An asymptotically distribution-free test for symmetry versus asymmetry. Journal of the American Statistical Association 75, 168–172] and the [Bai, J., and Ng, S., 2001. A consistent test for conditional symmetry in time series models. Journal of Econometrics 103, 225–258] test for conditional symmetry. Using data that spans over three decades, we find overwhelming evidence of conditional symmetry of residential and industrial electricity consumption. This finding implies that the use of econometric tests based on linear data generating processes is credible.

A Robust Entropy-Based Test of Asymmetry for Discrete and Continuous Processes

We consider a metric entropy capable of detecting deviations from symmetry that is suitable for both discrete and continuous processes. A test statistic is constructed from an integrated normed difference between nonparametric estimates of two density functions. The null distribution (symmetry) is obtained by resampling from an artificially lengthened series constructed from a rotation of the original series about its mean (median, mode). Simulations demonstrate that the test has correct size and good power in the direction of interesting alternatives, while applications to updated Nelson and Plosser (1982) data demonstrate its potential power gains relative to existing tests.

Investigating business cycle asymmetry for the G7 countries: Evidence from over a century of data

In this paper we test for asymmetric behaviour of business cycles for the G7 countries, using the entropy-based test for asymmetry suggested by Racine and Maasoumi [Racine, J.S., & Maasoumi, E. (in press-a). A versatile and robust metric entropy test of time-reversibility, and other hypotheses, Journal of Econometrics; Racine, J.S., & Maasoumi, E. (in press-b). A robust entropy based test for asymmetry. Econometric Reviews.]. We find overwhelming evidence of symmetry. In only 14% of the cases, we find some evidence of asymmetric behaviour of GDP and per capita GDP. More importantly, the period marked by the flexible exchange rate regime, over which much of the empirical work for the G7 countries has been conducted, evidence suggests that the two GDP series are symmetric.

Testing goodness-of-fit for Laplace distribution based on maximum entropy

This article presents the goodness-of-fit tests for the Laplace distribution based on its maximum entropy characterization result. The critical values of the test statistics estimated by Monte Carlo simulations are tabulated for various window and sample sizes. The test statistics use an entropy estimator depending on the window size; so, the choice of the optimal window size is an important problem. The window sizes for yielding the maximum power of the tests are given for selected sample sizes. Power studies are performed to compare the proposed tests with goodness-of-fit tests based on the empirical distribution function. Simulation results report that entropy-based tests have consistently higher power than EDF tests against almost all alternatives considered.

An Entropy-Based Statistic for Genomewide Association Studies

Efficient genotyping methods and the availability of a large collection of single-nucleotide polymorphisms provide valuable tools for genetic studies of human disease. The standard chi2 statistic for case-control studies, which uses a linear function of allele frequencies, has limited power when the number of marker loci is large. We introduce a novel test statistic for genetic association studies that uses Shannon entropy and a nonlinear function of allele frequencies to amplify the differences in allele and haplotype frequencies to maintain statistical power with large numbers of marker loci. We investigate the relationship between the entropy-based test statistic and the standard chi2 statistic and show that, in most cases, the power of the entropy-based statistic is greater than that of the standard chi2 statistic. The distribution of the entropy-based statistic and the type I error rates are validated using simulation studies. Finally, we apply the new entropy-based test statistic to two real data sets, one for the COMT gene and schizophrenia and one for the MMP-2 gene and esophageal carcinoma, to evaluate the performance of the new method for genetic association studies. The results show that the entropy-based statistic obtained smaller P values than did the standard chi2 statistic.

Asymptotic Distribution Theory for Nonparametric Entropy Measures of Serial Dependence

Entropy is a classical statistical concept with appealing properties. Establishing asymptotic distribution theory for smoothed nonparametric entropy measures of dependence has so far proved challenging. In this paper, we develop an asymptotic theory for a class of kernel-based smoothed nonparametric entropy measures of serial dependence in a time-series context. We use this theory to derive the limiting distribution of Granger and Lin's (1994) normalized entropy measure of serial dependence, which was previously not available in the literature. We also apply our theory to construct a new entropy-based test for serial dependence, providing an alternative to Robinson's (1991) approach. To obtain accurate inferences, we propose and justify a consistent smoothed bootstrap procedure. The naive bootstrap is not consistent for our test. Our test is useful in, for example, testing the random walk hypothesis, evaluating density forecasts, and identifying important lags of a time series. It is asymptotically locally more powerful than Robinson's (1991) test, as is confirmed in our simulation. An application to the daily S&P 500 stock price index illustrates our approach.