Neyman-Pearson Hypothesis Testing Research Articles

Optimal Automatic Wide-Area Discrimination of Fish Shoals from Seafloor Geology with Multi-Spectral Ocean Acoustic Waveguide Remote Sensing in the Gulf of Maine

Ocean Acoustic Waveguide Remote Sensing (OAWRS) enables fish population density distributions to be instantaneously quantified and continuously monitored over wide areas. Returns from seafloor geology can also be received as background or clutter by OAWRS when insufficient fish populations are present in any region. Given the large spatial regions that fish inhabit and roam over, it is important to develop automatic methods for determining whether fish are present at any pixel in an OAWRS image so that their population distributions, migrations and behaviour can be efficiently analyzed and monitored in large data sets. Here, a statistically optimal automated approach for distinguishing fish from seafloor geology in OAWRS imagery is demonstrated with Neyman–Pearson hypothesis testing which provides the highest true-positive classification rate for a given false-positive rate. Multispectral OAWRS images of large herring shoals during spawning migration to Georges Bank are analyzed. Automated Neyman-Pearson hypothesis testing is shown to accurately distinguish fish from seafloor geology through their differing spectral responses at any space and time pixel in OAWRS imagery. These spectral differences are most dramatic in the vicinity of swimbladder resonances of the fish probed by OAWRS. When such significantly different spectral dependencies exist between fish and geologic scattering, the approach presented provides an instantaneous, reliable and statistically optimal means of automatically distinguishing fish from seafloor geology at any spatial pixel in wide-area OAWRS images. Employing Kullback–Leibler divergence or the relative entropy in bits from Information Theory is shown to also enable automatic discrimination of fish from seafloor by their distinct statistical scattering properties across sensing frequency, but without the statistical optimal properties of the Neyman–Pearson approach.

A comparative study of frequentist vs Bayesian A/B testing in the detection of E-commerce fraud

PurposeThis paper tests whether Bayesian A/B testing yields better decisions that traditional Neyman-Pearson hypothesis testing. It proposes a model and tests it using a large, multiyear Google Analytics (GA) dataset.Design/methodology/approachThis paper is an empirical study. Competing A/B testing models were used to analyze a large, multiyear dataset of GA dataset for a firm that relies entirely on their website and online transactions for customer engagement and sales.FindingsBayesian A/B tests of the data not only yielded a clear delineation of the timing and impact of the intellectual property fraud, but calculated the loss of sales dollars, traffic and time on the firm’s website, with precise confidence limits. Frequentist A/B testing identified fraud in bounce rate at 5% significance, and bounces at 10% significance, but was unable to ascertain fraud at the standard significance cutoffs for scientific studies.Research limitations/implicationsNone within the scope of the research plan.Practical implicationsBayesian A/B tests of the data not only yielded a clear delineation of the timing and impact of the IP fraud, but calculated the loss of sales dollars, traffic and time on the firm’s website, with precise confidence limits.Social implicationsBayesian A/B testing can derive economically meaningful statistics, whereas frequentist A/B testing only providep-value’s whose meaning may be hard to grasp, and where misuse is widespread and has been a major topic in metascience. While misuse ofp-values in scholarly articles may simply be grist for academic debate, the uncertainty surrounding the meaning ofp-values in business analytics actually can cost firms money.Originality/valueThere is very little empirical research in e-commerce that uses Bayesian A/B testing. Almost all corporate testing is done via frequentist Neyman-Pearson methods.

A New Induced GNSS Spoofing Detection Method Based on Weighted Second-Order Central Moment

Global navigation satellite system (GNSS) spoofing causes the victim receiver to deduce false positioning and timing data; this notably threatens navigational safety. Thus, anti-spoofing techniques that improve the reliability of GNSS systems, for which interference detection is critical, are essential. Based on the distortion of tracking loop correlation function symmetry of the target receiver caused by gradual adjustment of induced spoofing signals, we proposed a new induced spoofing detection method that uses the weighted second-order central moment (WSCM) difference in the time-domain transient response of multiple correlators of the left and right peaks to obtain the test statistic, theoretically proving that the test statistic follows Gaussian distribution. The Neyman-Pearson hypothesis test method is used to determine the optimal test threshold and determine whether the receiver is being spoofed. The proposed WSCM-based method for spoofing detection was compared with three conventional methods in Scenarios 4 and 7 of the Texas Spoofing Test Battery database, showing that the detection probability of the proposed method is at least 24.15% higher at a false alarm rate of 10% and is more advantageous at lower false alarm rates and the alert time is shortened by at least 30 seconds, enabling at least a 20% faster detection efficiency. The proposed method overcomes the problem of existing methods, which are associated with difficulties in capturing the subtle time-varying effects of the relative carrier phase between the spoofing and authentic signals; thus, it provides excellent detection accuracy and effectiveness, showing broad potential applicability in GNSS spoofing detection.

Analytical Model for Mitigating Primary User Emulation Attack using Hypothesis Testing in Cognitive Radio Networks

The Cognitive Radio (CR) is the key technology to deal with spectrum scarcity by allowing unlicensed CR users to coexist with existing users in licensed spectrum bands without interfering with binding communications. Cognitive technology provides the spectrum ability to be shared with licensed networks. This radio system can adjust its transmitter and recipient parameters on the basis of interaction with the current conditions in the environment. Due to this flexibility cognitive radios are exploring different types of threats and security attacks. In this paper, we focus primarily on the Primary User Emulation Attack (PUEA) which is a major attack in the cognitive radio network. PUEA is similar to a Denial-of-service attack that can seriously disrupt the spectrum sensing process and deny other legitimate secondary users to access the available spectrum. We proposed a Neyman-Pearson composite hypothesis test (NPCHT) based analytical model to study the impact of PUEA in a fading wireless communication environment. Simulation results show that the proposed techniques can substantially reduce the impact of the malicious attack on the network

Neyman-Pearson Hypothesis Testing, Epistemic Reliability and Pragmatic Value-Laden Asymmetric Error Risks

We show that if among the tested hypotheses the number of true hypotheses is not equal to the number of false hypotheses, then Neyman-Pearson theory of testing hypotheses does not warrant minimal epistemic reliability (the feature of driving to true conclusions more often than to false ones). We also argue that N-P does not protect from the possible negative effects of the pragmatic value-laden unequal setting of error probabilities on N-P’s epistemic reliability. Most importantly, we argue that in the case of a negative impact no methodological adjustment is available to neutralize it, so in such cases the discussed pragmatic-value-ladenness of N-P inevitably compromises the goal of attaining truth.

Errors in Statistical Inference Under Model Misspecification: Evidence, Hypothesis Testing, and AIC.

The methods for making statistical inferences in scientific analysis have diversified even within the frequentist branch of statistics, but comparison has been elusive. We approximate analytically and numerically the performance of Neyman-Pearson hypothesis testing, Fisher significance testing, information criteria, and evidential statistics (Royall, 1997). This last approach is implemented in the form of evidence functions: statistics for comparing two models by estimating, based on data, their relative distance to the generating process (i.e., truth) (Lele, 2004). A consequence of this definition is the salient property that the probabilities of misleading or weak evidence, error probabilities analogous to Type 1 and Type 2 errors in hypothesis testing, all approach 0 as sample size increases. Our comparison of these approaches focuses primarily on the frequency with which errors are made, both when models are correctly specified, and when they are misspecified, but also considers ease of interpretation. The error rates in evidential analysis all decrease to 0 as sample size increases even under model misspecification. Neyman-Pearson testing on the other hand, exhibits great difficulties under misspecification. The real Type 1 and Type 2 error rates can be less, equal to, or greater than the nominal rates depending on the nature of model misspecification. Under some reasonable circumstances, the probability of Type 1 error is an increasing function of sample size that can even approach 1! In contrast, under model misspecification an evidential analysis retains the desirable properties of always having a greater probability of selecting the best model over an inferior one and of having the probability of selecting the best model increase monotonically with sample size. We show that the evidence function concept fulfills the seeming objectives of model selection in ecology, both in a statistical as well as scientific sense, and that evidence functions are intuitive and easily grasped. We find that consistent information criteria are evidence functions but the MSE minimizing (or efficient) information criteria (e.g., AIC, AICc, TIC) are not. The error properties of the MSE minimizing criteria switch between those of evidence functions and those of Neyman-Pearson tests depending on models being compared.

Cross-Level Detection Framework for Attacks on Cyber-Physical Systems

Anomaly detection is critical in thwarting malicious attacks on Cyber-Physical Systems. This work presents a novel inference engine that integrates two heterogeneous anomaly detectors, working at different levels of the system architecture, in order to produce a cross-level detector more effective than either one separately. The macro- or process-level detector uses a bank of observers of the physical plant that estimate the state of the process suspected to be under attack, specifically for its sensor to be compromised, from data gathered by available networked sensors. The estimates are then combined using a consensus algorithm to determine if the suspect sensor is reporting false readings. The micro-level detector uses time-sampled side-channel power measurements of an integrated circuit on the suspect sensor. By comparing power measurements against those from a known good state, differences indicate the code running inside has been altered. The cross-level detector performs a two-dimensional Neyman-Pearson hypothesis test that declares the presence of an attack on the sensor node. The cross-level detector is shown to be more accurate and less latent than its constituent parts. Detection was tested against a range of False Data Injection attacks on a hardware prototype and the detector performance was measured experimentally. The cross-level detector on average achieved a 93% rate of correct detection, compared with 72 and 85% for the macro- and micro-level detectors, respectively; and a 50% reduction in latency compared to the macro-level detector.

Determining the probability of correct resolution of the left–right ambiguity in towed array sonar

When a towed sonar array is straight, it has the difficulty that it cannot distinguish a contact on the left from one at the same angle on the right. When the array isnotstraight and its shape known, we calculate the probability that the left–right ambiguity is resolved correctly, using the Neyman–Pearson hypothesis testing framework, assuming a delay-sum beamformer, a single-frequency contact, and Gaussian noise. We also initially consider the noise field to be uncorrelated and show that the evaluation of the probability of correct resolution reduces to evaluating a one-dimensional integral. We find, as expected, that the probability increases, as the signal-to-noise ratio and the lateral deviation of the array from straight increase. For demonstration purposes, we also evaluate the probability of correct resolution for two representative shapes the array might assume in practice. Finally, we consider a more realistic, correlated noise field and we show that the initial assumption of an uncorrelated noise field provides a good approximation when the lateral deviation of the array is sufficiently large.

Tally-Based Simple Decoders for Traitor Tracing and Group Testing

The topic of this paper is collusion resistant watermarking, also known as traitor tracing, in particular bias-based traitor tracing codes as introduced by Tardos. The past years have seen an ongoing effort to construct efficient high-performance decoders for these codes. In this paper we construct a score system from the Neyman-Pearson hypothesis test (which is known to be the most powerful test possible) into which we feed more evidence than in previous work, in particular the symbol tallies for all columns of the code matrix. As far as we know, until now simple decoders using Neyman-Pearson have taken into consideration only the codeword of a single user, namely the user under scrutiny. The Neyman–Pearson score needs as input the attack strategy of the colluders, which typically is not known to the tracer. We insert the interleaving attack, which plays a very special role in the theory of bias-based traitor tracing by virtue of being part of the asymptotic (i.e., large coalition size) saddle-point solution. The score system obtained in this way is universal: effective not only against the interleaving attack, but against all other attack strategies as well. Our score function for one user depends on the other users’ codewords in a very simple way through the symbol tallies, which are easily computed. We present bounds on the false positive probability and show receiver operating characteristic curves obtained from simulations. We investigate the probability distribution of the score. Finally, we apply our construction to the area of (medical) group testing, which is related to traitor tracing.

A New Strategy for Testing Structural Equation Models

One of the most important issues in structural equation modeling concerns testing model fit. We propose to retain the likelihood ratio test in combination with decision criteria that increase with sample size. Specifically, rooted in Neyman–Pearson hypothesis testing, we advocate balancing α- and β-error risks. This strategy has a number of desirable consequences and addresses several objections that have been raised against the likelihood ratio test in model evaluation. First, balancing error risks avoids logical problems with Fisher-type hypotheses tests when predicting the null hypothesis (i.e., model fit). Second, both types of statistical decision errors are controlled. Third, larger samples are encouraged (rather than penalized) because both error risks diminish as the sample size increases. Finally, the strategy addresses the concern that structural equation models cannot necessarily be expected to provide an exact description of real-world phenomena.

Hypothesis testing and decision theoretic approach for fault detection in wireless sensor networks

Sensor networks aim at monitoring their surroundings for event detection and object tracking. In practice, due to failure or death of sensors, false signal can be transmitted. In this article, we consider the problem of fault detection in wireless sensor networks, in particular, addressing both the noise-related measurement error and sensor fault simultaneously in fault detection. We assume that the sensors are placed at the centre of a square (or hexagonal) cell in the region of interest (ROI) and, if the event occurs, it occurs at a particular cell of the ROI. We propose fault detection schemes that consider error probabilities in the optimal event detection process. We develop the schemes under the consideration of Neyman–Pearson hypothesis test and Bayes test.

Hyperspectral Data Geometry-Based Estimation of Number of Endmembers Using p-Norm-Based Pure Pixel Identification Algorithm

Hyperspectral endmember extraction is a process to estimate endmember signatures from the hyperspectral observations, in an attempt to study the underlying mineral composition of a landscape. However, estimating the number of endmembers, which is usually assumed to be known a priori in most endmember estimation algorithms (EEAs), still remains a challenging task. In this paper, assuming hyperspectral linear mixing model, we propose a hyperspectral data geometry-based approach for estimating the number of endmembers by utilizing successive endmember estimation strategy of an EEA. The approach is fulfilled by two novel algorithms, namely geometry-based estimation of number of endmembers—convex hull (GENE-CH) algorithm and affine hull (GENE-AH) algorithm. The GENE-CH and GENE-AH algorithms are based on the fact that all the observed pixel vectors lie in the convex hull and affine hull of the endmember signatures, respectively. The proposed GENE algorithms estimate the number of endmembers by using the Neyman–Pearson hypothesis testing over the endmember estimates provided by a successive EEA until the estimate of the number of endmembers is obtained. Since the estimation accuracies of the proposed GENE algorithms depend on the performance of the EEA used, a reliable, reproducible, and successive EEA, called <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$p$</tex></formula> -norm-based pure pixel identification (TRI-P) algorithm is then proposed. The performance of the proposed TRI-P algorithm, and the estimation accuracies of the GENE algorithms are demonstrated through Monte Carlo simulations. Finally, the proposed GENE and TRI-P algorithms are applied to real AVIRIS hyperspectral data obtained over the Cuprite mining site, Nevada, and some conclusions and future directions are provided.

Monetary policy and inferential expectations of exchange rates

We present a macroeconomic market experiment to isolate the impact of monetary shocks on the exchange rate, as an alternative to SVAR identification. In a non-stochastic treatment, covered interest rate parity holds and predicted exchange rates are tracked well. In a stochastic treatment, we model expectations using a Neyman–Pearson hypothesis test (inferential expectations) and find evidence of belief conservatism and uncovered interest rate parity failure. The market environment magnifies belief conservatism, which is opposite to the standard claim that markets tend to eliminate individual choice anomalies.

Applications of predictive maintenance techniques in industrial systems

Prognostic methods represent a new methodology for system maintenance which offers significant time and cost savings. The paper offers a short overview of the available prognosis techniques and proposes the implementation of one model-based and one data-driven method. As a representative of the model-based methods the autoregressive moving average (ARMA) modeling approach is chosen. The estimated model parameters are further used for implementing the early change detector which is realized as a Neyman-Pearson hypothesis test. On the other hand, hidden Markov model (HMM) based prognosis illustrates the use of data-driven techniques. Using the cross-correlation input-output functions, HMM prognosis algorithm is proposed, as a suitable way of timely detection. Both techniques were implemented to detect performance changes of the water level sensor in a steam separator system in thermal power plants.

Locating anomalies by gravity gradiometry using the matched filter with a probabilistic assessment

The matched filter can be used to identify the location of a specific signal embedded in background correlated noise. The maximum filter output indicates the likely location and, with appropriate statistical assumptions on the noise, it also serves as a test statistic in the probabilistic evaluation of the filter’s performance. Different setups of the Neyman-Pearson statistical hypothesis test yield predictions of either the probability of a miss or that of a false alarm. The needed statistics of the maximum filter output are properly obtained using the distribution of order statistics. Through Monte Carlo simulations, we analyzed the ability of the matched filter to identify a sub-surface anomaly in typically correlated gravity fields using observations of elements of the gravity gradient tensor. We also evaluated the reliability of the hypothesis testing and the associated predicted probabilities of misses and false alarms. Our simulations and statistical analyses confirm that the power of the tests increases as the signal strength increases and as more gradient tensor components per observation point are included. We also found that the hypothesis test that is designed to predict the probability of a miss is more robust and powerful than the one for predicting a false alarm. Moreover, the probability of a miss is somewhat smaller than the probability of a false alarm under otherwise equal circumstances.

A Bayesian Framework for Collaborative Multi-Source Signal Sensing

This paper introduces a Bayesian framework to detect multiple signals embedded in noisy observations, from an array of sensors. For various states of knowledge on the communication channel and the noise at the receiving sensors, a marginalization procedure based on random matrix theory techniques, in conjunction with the maximum entropy principle, is used to compute the Neyman-Pearson hypothesis testing criterion. Quite remarkably, although rather involved, explicit expressions for the Bayesian detector are derived which enable to decide on the presence of signal sources in a noisy wireless environment. Under the hypotheses that the true channel conditions adhere the maximum entropy model, the proposed detector is the optimal Neyman-Pearson detector; if so, the performance of the derived decision criteria can be used as an upper bound for the performance of alternative detectors. In particular, simulation results are provided that suggest that the classical energy detector is close-to-optimal when the noise power is a priori known to the sensor array, especially when many sources simultaneously transmit, while the conditioning number-based detector, used classically when the noise power is unknown, is shown to perform poorly in comparison to the proposed optimal detector.

Performance issues in biometric authentication based on information theoretic concepts: A review

Many of the performance evaluation techniques for biometric authentication use error probabilities to yield a measure called receiver operating characteristic (ROC). The ROC is based on the Neyman-Pearson hypothesis testing and is obtained by varying a threshold for decision making. This measure is dependent on database partitioning and choice of thresholds. Also, obtaining the probability distributions, and thus the ROC, is computationally complex. Recent approaches based on information theoretic models partially overcome these limitations and also provide insight into the performance of biometric authentication techniques. Measures in line with obtaining Chernoff capacity and with Shannon capacity have been proposed, and are respectively called recognition capacity and constrained capacity. Measures which are largely independent of data size and quality are based on the minimization of false matches between templates and are good indicators of biometrics uniqueness or, equivalently, its random correspondence. The parameters related to confidence intervals are however obtained empirically. One such measure is obtained from the probability distribution of the Hamming distance between templates of the iris biometrics. Also, relative entropy of features between a user and the population yields a measure of uniqueness for face biometrics. Another approach is to measure the probability of this random correspondence (PRC). PRC of a fingerprint biometrics has been obtained using compound statistical distributions. Another formulation of PRC is based on a rate-distortion framework by applying a distortion constraint to a codebook of binarized features. Biometric templates are akin to noisy source symbols in an information theoretic setup. A bound on the PRC has been obtained by developing error exponents of noisy biometrics represented in terms of a binary source-channel model. The method has low computational complexity, is not limited to a specific biometrics, and does not require empirically obtained confidence intervals.

Optimal Noise Benefits in Neyman–Pearson and Inequality-Constrained Statistical Signal Detection

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> We present theorems and an algorithm to find optimal or near-optimal “stochastic resonance” (SR) noise benefits for Neyman–Pearson hypothesis testing and for more general inequality-constrained signal detection problems. The optimal SR noise distribution is just the randomization of two noise realizations when the optimal noise exists for a single inequality constraint on the average cost. The theorems give necessary and sufficient conditions for the existence of such optimal SR noise in inequality-constrained signal detectors. There exists a sequence of noise variables whose detection performance limit is optimal when such noise does not exist. Another theorem gives sufficient conditions for SR noise benefits in Neyman–Pearson and other signal detection problems with inequality cost constraints. An upper bound limits the number of iterations that the algorithm requires to find near-optimal noise. The appendix presents the proofs of the main results. </para>

Validation of null model tests using Neyman–Pearson hypothesis testing theory

A long-standing question in ecology is whether interspecific competition affects co-occurrence patterns of species. Null model tests of presence–absence data (NMTPAs) constitute an important approach to address the question, but different tests often give conflicting results when applied to the same data. Neyman–Pearson hypothesis testing theory provides a rigorous and well accepted framework for assessing the validity and optimality of statistical tests. Here, I treat NMTPAs within this framework, and measure the robustness and bias of 72 representative tests. My results indicate that, when restrictive assumptions are met, existing NMTPAs are adequate, but for general testing situations, the use of all existing NMTPAs is unjustified — the tests are nonrobust or biased. For many current applications of NMTPAs, restrictive assumptions appear unmet, so these results illustrate an area in which existing NMTPAs can be improved. In addition to highlighting useful improvements to existing NMTPAs, the results here provide a rigorous framework for developing improved methods.

Statistical Issues in Ecological Risk Assessment

ABSTRACT Ecological risk assessment (ERA) is concerned with making decisions about the natural environment under uncertainty. Statistical methodology provides a natural framework for risk characterization and manipulation with many quantitative ERAs relying heavily on Neyman-Pearson hypothesis testing and other frequentist modes of inference. Bayesian statistical methods are becoming increasingly popular in ERA as they are seen to provide legitimate ways of incorporating subjective belief or expert opinion in the form of prior probability distributions. This article explores some of the concepts, strengths and weaknesses, and difficulties associated with both paradigms. The main points are illustrated with an example of setting a risk-based “trigger” level for uranium concentrations in the Magela Creek catchment of the Northern Territory of Australia.