Neyman-Pearson Test Research Articles

Goodness of fit by Neyman-Pearson testing

The Neyman-Pearson strategy for hypothesis testing can be employed for goodness of fit if the alternative hypothesis is selected from data by exploring a rich parametrised family of models, while controlling the impact of statistical fluctuations. The New Physics Learning Machine (NPLM) methodology has been developed as a concrete implementation of this idea, to target the detection of new physical effects in the context of high energy physics collider experiments. In this paper we conduct a comparison of this approach to goodness of fit with others, in particular with classifier-based strategies that share strong similarities with NPLM. From our comparison, NPLM emerges as the more sensitive test to small departures of the data from the expected distribution and not biased towards detecting specific types of anomalies. These features make it suited for agnostic searches for new physics at collider experiments. Its deployment in other scientific and industrial scenarios should be investigated.

Performance Analysis of Multiple-Antenna Ambient Backscatter Systems at Finite Blocklengths

This paper analyzes the maximal achievable rate for a given blocklength and maximal error probability over a multiple-antenna ambient backscatter channel. The result consists of a finite blocklength channel coding achievability bound and a converse bound for the legacy system with finite alphabet constraints and multiple-input-multiple-output based on the Neyman-Pearson test, the Berry-Esseen theorem, and the Mellin transform. Then, we derive the closed-form expression of the mutual information and the information variance to reduce the complexity of the computation. By applying the low-complexity ML detection, the relation between the maximal error probability of the RF source signal and the average error probability of the tag symbol with respect to the blocklength is proposed. Finally, numerical evaluation of these bounds shows fast convergence to the maximal achievable rate as the blocklength increases and also proves that the information variance is an accurate measure of the backoff from the maximal achievable rate due to finite blocklength.

Sensor fault detection of vehicle suspension systems based on transmissibility operators and Neyman–Pearson test

Sensor fault detection of vehicle suspension systems based on transmissibility operators and Neyman–Pearson test

Sampling distribution for single-regression Granger causality estimators

Summary The single-regression Granger–Geweke causality estimator has previously been shown to solve known problems associated with the more conventional likelihood ratio estimator; however, its sampling distribution has remained unknown. We show that, under the null hypothesis of vanishing Granger causality, the single-regression estimator converges to a generalized χ2 distribution, which is well approximated by a Γ distribution. We show that this holds too for Geweke’s spectral causality averaged over a given frequency band, and derive explicit expressions for the generalized χ2 and Γ-approximation parameters in both cases. We present a Neyman–Pearson test based on the single-regression estimators, and discuss how it may be deployed in empirical scenarios. We outline how our analysis may be extended to the conditional case, point-frequency spectral Granger causality and the important case of state-space Granger causality.

Optimal Automatic Wide-Area Discrimination of Fish Shoals from Seafloor Geology with Multi-Spectral Ocean Acoustic Waveguide Remote Sensing in the Gulf of Maine

Ocean Acoustic Waveguide Remote Sensing (OAWRS) enables fish population density distributions to be instantaneously quantified and continuously monitored over wide areas. Returns from seafloor geology can also be received as background or clutter by OAWRS when insufficient fish populations are present in any region. Given the large spatial regions that fish inhabit and roam over, it is important to develop automatic methods for determining whether fish are present at any pixel in an OAWRS image so that their population distributions, migrations and behaviour can be efficiently analyzed and monitored in large data sets. Here, a statistically optimal automated approach for distinguishing fish from seafloor geology in OAWRS imagery is demonstrated with Neyman–Pearson hypothesis testing which provides the highest true-positive classification rate for a given false-positive rate. Multispectral OAWRS images of large herring shoals during spawning migration to Georges Bank are analyzed. Automated Neyman-Pearson hypothesis testing is shown to accurately distinguish fish from seafloor geology through their differing spectral responses at any space and time pixel in OAWRS imagery. These spectral differences are most dramatic in the vicinity of swimbladder resonances of the fish probed by OAWRS. When such significantly different spectral dependencies exist between fish and geologic scattering, the approach presented provides an instantaneous, reliable and statistically optimal means of automatically distinguishing fish from seafloor geology at any spatial pixel in wide-area OAWRS images. Employing Kullback–Leibler divergence or the relative entropy in bits from Information Theory is shown to also enable automatic discrimination of fish from seafloor by their distinct statistical scattering properties across sensing frequency, but without the statistical optimal properties of the Neyman–Pearson approach.

Optimal Generic L-Out-of-M Counting Rule for Neyman-Pearson Test in Cognitive Radio Networks

Optimal Generic L-Out-of-M Counting Rule for Neyman-Pearson Test in Cognitive Radio Networks

A comparative study of frequentist vs Bayesian A/B testing in the detection of E-commerce fraud

PurposeThis paper tests whether Bayesian A/B testing yields better decisions that traditional Neyman-Pearson hypothesis testing. It proposes a model and tests it using a large, multiyear Google Analytics (GA) dataset.Design/methodology/approachThis paper is an empirical study. Competing A/B testing models were used to analyze a large, multiyear dataset of GA dataset for a firm that relies entirely on their website and online transactions for customer engagement and sales.FindingsBayesian A/B tests of the data not only yielded a clear delineation of the timing and impact of the intellectual property fraud, but calculated the loss of sales dollars, traffic and time on the firm’s website, with precise confidence limits. Frequentist A/B testing identified fraud in bounce rate at 5% significance, and bounces at 10% significance, but was unable to ascertain fraud at the standard significance cutoffs for scientific studies.Research limitations/implicationsNone within the scope of the research plan.Practical implicationsBayesian A/B tests of the data not only yielded a clear delineation of the timing and impact of the IP fraud, but calculated the loss of sales dollars, traffic and time on the firm’s website, with precise confidence limits.Social implicationsBayesian A/B testing can derive economically meaningful statistics, whereas frequentist A/B testing only providep-value’s whose meaning may be hard to grasp, and where misuse is widespread and has been a major topic in metascience. While misuse ofp-values in scholarly articles may simply be grist for academic debate, the uncertainty surrounding the meaning ofp-values in business analytics actually can cost firms money.Originality/valueThere is very little empirical research in e-commerce that uses Bayesian A/B testing. Almost all corporate testing is done via frequentist Neyman-Pearson methods.

A New Induced GNSS Spoofing Detection Method Based on Weighted Second-Order Central Moment

Global navigation satellite system (GNSS) spoofing causes the victim receiver to deduce false positioning and timing data; this notably threatens navigational safety. Thus, anti-spoofing techniques that improve the reliability of GNSS systems, for which interference detection is critical, are essential. Based on the distortion of tracking loop correlation function symmetry of the target receiver caused by gradual adjustment of induced spoofing signals, we proposed a new induced spoofing detection method that uses the weighted second-order central moment (WSCM) difference in the time-domain transient response of multiple correlators of the left and right peaks to obtain the test statistic, theoretically proving that the test statistic follows Gaussian distribution. The Neyman-Pearson hypothesis test method is used to determine the optimal test threshold and determine whether the receiver is being spoofed. The proposed WSCM-based method for spoofing detection was compared with three conventional methods in Scenarios 4 and 7 of the Texas Spoofing Test Battery database, showing that the detection probability of the proposed method is at least 24.15% higher at a false alarm rate of 10% and is more advantageous at lower false alarm rates and the alert time is shortened by at least 30 seconds, enabling at least a 20% faster detection efficiency. The proposed method overcomes the problem of existing methods, which are associated with difficulties in capturing the subtle time-varying effects of the relative carrier phase between the spoofing and authentic signals; thus, it provides excellent detection accuracy and effectiveness, showing broad potential applicability in GNSS spoofing detection.

Optimal and Maximin Procedures for Multiple Testing Problems

AbstractMultiple testing problems (MTPs) are a staple of modern statistical analysis. The fundamental objective of MTPs is to reject as many false null hypotheses as possible (that is, maximize some notion of power), subject to controlling an overall measure of false discovery, like family-wise error rate (FWER) or false discovery rate (FDR). In this paper we provide generalizations to MTPs of the optimal Neyman-Pearson test for a single hypothesis. We show that for simple hypotheses, for both FWER and FDR and relevant notions of power, finding the optimal multiple testing procedure can be formulated as infinite dimensional binary programs and can in principle be solved for any number of hypotheses. We also characterize maximin rules for complex alternatives, and demonstrate that such rules can be found in practice, leading to improved practical procedures compared to existing alternatives that guarantee strong error control on the entire parameter space. We demonstrate the usefulness of these novel rules for identifying which studies contain signal in numerical experiments as well as in application to clinical trials with multiple studies. In various settings, the increase in power from using optimal and maximin procedures can range from 15% to more than 100%.

Low-Complexity Distributed Detection With One-Bit Memory Under Neyman–Pearson Criterion

We consider a multi-stage distributed detection scenario, where $n$ sensors and a fusion center (FC) are deployed to accomplish a binary hypothesis test. At each time stage, local sensors generate binary messages, assumed to be spatially and temporally independent given the hypothesis, and then upload them to the FC for global detection decision making. We suppose a one-bit memory is available at the FC to store its decision history and focus on developing iterative fusion schemes. We first visit the detection problem of performing the Neyman-Pearson (N-P) test at each stage and give an optimal algorithm, called the oracle algorithm, to solve it. Structural properties and limitation of the fusion performance in the asymptotic regime are explored for the oracle algorithm. We notice the computational inefficiency of the oracle fusion and propose a low-complexity alternative, for which the likelihood ratio (LR) test threshold is tuned in connection to the fusion decision history compressed in the one-bit memory. The low-complexity algorithm greatly brings down the computational complexity at each stage from $O(4^n)$ to $O(n)$. We show that the proposed algorithm is capable of converging exponentially to the same detection probability as that of the oracle one. Moreover, the rate of convergence is shown to be asymptotically identical to that of the oracle algorithm. Finally, numerical simulations and real-world experiments demonstrate the effectiveness and efficiency of our distributed algorithm.

A Hybrid Monitoring Procedure for Detecting Abnormality with Application to Energy Consumption Data

The complexity of energy infrastructure at large institutions increasingly calls for data-driven monitoring of energy usage. This article presents a hybrid monitoring algorithm for detecting consumption surges using statistical hypothesis testing, leveraging the posterior distribution and its information about uncertainty to introduce randomness in the parameter estimates, while retaining the frequentist testing framework. This hybrid approach is designed to be asymptotically equivalent to the Neyman-Pearson test. We show via extensive simulation studies that the hybrid approach enjoys control over type-1 error rate even with finite sample sizes whereas the naive plug-in method tends to exceed the specified level, resulting in overpowered tests. The proposed method is applied to the natural gas usage data at the University of Connecticut.

On statistical Multi-Objective optimization of sensor networks and optimal detector derivation for structural health monitoring

Sensor placement and structural health classifiers are fundamental components of Structural Health Monitoring (SHM) systems, as they largely define system detection (or classification) performance. Optimal sensor placement strategies are designed to maximize the ability to detect damage or to minimize lifetime costs, given limited resource availability. However, usually choosing one strategy over the other and non-optimal detector implementation may provide poorly performing solutions in terms of detection performance or total cost, even though both are critical objectives for a cost-effective SHM system implementation.The work proposes a unique and coherent framework for optimal detector and sensing network design for SHM. After an optimal detector is defined based on the Neyman-Pearson likelihood ratio test, classification performance indexes are used in a multi-objective optimization paradigm for optimal sensor placement. Specifically, the optimization considers maximizing the classification performances and, simultaneously, minimizing a measure of total cost or risk in a Bayesian sense.Even though the approach is general for any structure and sensor measurement process, the method is numerically verified with a cracked plate under tension and monitored by measurements of local strain serving as the surrogate SHM system. The results are also validated by comparing the multi-objective optimal design to engineering judgment and single-objective-based solutions in terms of probability of detection and costs. The advantages of an optimization scheme are emphasized with respect to an engineering scheme and, above all, how a multi-objective optimization strategy reflects a conjunct saving in costs and improvement in detection performances.

Analytical Model for Mitigating Primary User Emulation Attack using Hypothesis Testing in Cognitive Radio Networks

The Cognitive Radio (CR) is the key technology to deal with spectrum scarcity by allowing unlicensed CR users to coexist with existing users in licensed spectrum bands without interfering with binding communications. Cognitive technology provides the spectrum ability to be shared with licensed networks. This radio system can adjust its transmitter and recipient parameters on the basis of interaction with the current conditions in the environment. Due to this flexibility cognitive radios are exploring different types of threats and security attacks. In this paper, we focus primarily on the Primary User Emulation Attack (PUEA) which is a major attack in the cognitive radio network. PUEA is similar to a Denial-of-service attack that can seriously disrupt the spectrum sensing process and deny other legitimate secondary users to access the available spectrum. We proposed a Neyman-Pearson composite hypothesis test (NPCHT) based analytical model to study the impact of PUEA in a fading wireless communication environment. Simulation results show that the proposed techniques can substantially reduce the impact of the malicious attack on the network

Nested code division multiple access for distance measuring equipment interference mitigation in L‐band digital aeronautical communication system

SummaryAir‐to‐ground (A/G) communications have been transferred from the congested very high frequency (VHF) band to occupy the guard bands between adjacent distance measurement equipment (DME) navigation channels within the L‐Band. However, the inlayed channels, called L‐band digital aeronautical communication system (LDACS) channels, experience severe interference with the coexisting DME channels. In this paper, a DME interference detection and mitigation technique is proposed to improve the performance of the LDACS in the presence of DME interference. The DME energy detector based on Neyman–Pearson test is proposed, which has more advantages than the other techniques due to its noncoherent nature and simplicity. Furthermore, the proposed DME detection technique is performed by using an adaptive threshold to achieve the best trade‐off between the DME signal detection and false alarm. The proposed DME interference mitigation technique relies on the proposed concept of nested RF upconversion, where the in‐phase and the quadrature components of an LDACS signal are forwarded to a code division multiple access (CDMA) block in which they are transmitted only over the quadrature branch once DME interfering signal is detected. In this work, the closed form expression for the probability of DME detection, false alarm, and the detection error rate are derived. Simulation is performed, and the results show that the proposed technique provides improvement in the probability of DME detection and the detection error rate compared to the previous work. Also, simulation results shows that the proposed technique provides lower bit error rate and less complexity compared to the previous work.

Mitigation Techniques for Puea In Cognitive Radio Networks Using Hypothesis Testing

This paper detects primary user emulation attacks in cognitive radio networks using Neyman-Pearson composite hypothesis test and Wald’s sequential probability ratio test. Most approaches in the literature on Primary user emulation attacks assume the presence of underlying sensor networks for localization of the malicious nodes. There are no analytical studies available in the literature to study Primary user emulation attacks in the presence of multiple malicious users in fading wireless environments. An NPCHT and WSPRT based analysis to detect Primary user emulation attack in fading wireless channels in the presence of multiple randomly located malicious users. It shows there is a range of network radii in which PUEA is most successful. The results also show that for the same desired threshold on the probability of missing the primary, WSPRT can achieve a successful probability PUEA 50% less than that obtained by NPCHT.

Performance Metrics in Cognitive Radio Networks

In Cognitive Radio Networks (CRN), the main aim is to allow the secondary users (SUs) to identify the empty bands and use them to transmit or receive data opportunistically. Primary users (PUs) have the priority to use a channel, while the secondary users must vacant this channel once a primary user requests it. An attack known in cognitive radio networks as a Primary User Emulation Attack (PUEA) aims to prevent the SU from using the empty bands. In this paper, an analytical and experimental approach is presented to mitigate the PUEA. This approach is based on obtaining the Probability Density Functions (PDFs) of the received powers at the secondary users from malicious nodes and also from the primary transmitter in the cognitive network. Then, these obtained PDFs are used in Neyman-Pearson composite hypothesis test to measure the performance metrics (probability of false alarm and miss detection in the network). The results proved that the performance metrics were greatly influenced by the network area, where the secondary user is located, and the threshold value λ used in the decision rule. Also, there are boundaries for the λ choices that cannot be overtaken.

Neyman-Pearson Hypothesis Testing, Epistemic Reliability and Pragmatic Value-Laden Asymmetric Error Risks

We show that if among the tested hypotheses the number of true hypotheses is not equal to the number of false hypotheses, then Neyman-Pearson theory of testing hypotheses does not warrant minimal epistemic reliability (the feature of driving to true conclusions more often than to false ones). We also argue that N-P does not protect from the possible negative effects of the pragmatic value-laden unequal setting of error probabilities on N-P’s epistemic reliability. Most importantly, we argue that in the case of a negative impact no methodological adjustment is available to neutralize it, so in such cases the discussed pragmatic-value-ladenness of N-P inevitably compromises the goal of attaining truth.

Mutual interference alignment for co-existing radar and communication systems

In this paper, we propose a mutual interference alignment (IA) method for interference elimination in co-existing multiple-input multiple-output (MIMO) radar and multi-user MIMO communication systems, namely, radar-communication systems. Traditional IA methods proposed for communication systems cannot be directly adopted in the radar-communications due to failure to fulfill the requirements of radar target detection. In contrast, our proposed mutual IA method can efficiently improve the signal-to-noise ratio (SNR) needed by radar functionalities without reducing the degrees-of-freedom (DoFs) of communication system, which implies no degradation for the communication performance. The algorithm idea is first to align all interferences at every receiver to one signal space, then the interferences can be easily canceled by receive filter. It is examined by the Neyman-Pearson (NP) test for radar functionality, while the communication performance is evaluated by DoFs and bit error rate (BER) analysis. Both theory analysis and numerical results are provided to validate the effectiveness of the proposed mutual IA method for co-existing radar and communication systems.

Coherent track‐before‐detect with micro‐Doppler signature estimation in array radars

In this work, the authors consider joint micro-Doppler signature estimation and track-before-detect for detecting and classifying manoeuvring and small rotary-wing aircraft with a phased array radar receiver. Such aircraft induce low signal-to-noise ratio reflections which complicates their detection. These difficulties are exacerbated by the high manoeuvrability of such aircraft which results with resolution cell migrations in conventional detectors. The authors propose a novel track-before-detect algorithm which jointly estimates the micro-Doppler signature, dominant reflectivity and kinematic trajectory of an object before the decision on its existence is made. These quantities are used in a Neyman-Pearson test against a constant false alarm rate threshold which is equivalent to evaluating a coherent long time integration value. The authors demonstrate the efficacy of the proposed scheme in a real data experiment and compare it with conventional techniques.

A Framework for Robust Assimilation of Potentially Malign Third-Party Data, and Its Statistical Meaning

This paper presents a model-based method for fusing data from multiple sensors with a hypothesis-test-based component for rejecting potentially faulty or otherwise malign data. Our framework is based on an extension of the classic particle filter algorithm for real-time state estimation of uncertain systems with nonlinear dynamics with partial and noisy observations. This extension, based on classical statistical theories, utilizes statistical tests against the system's observation model. We discuss the application of the two major statistical testing frameworks, Fisherian significance testing and Neyman-Pearsonian hypothesis testing, to the Monte Carlo and sensor fusion settings. The Monte Carlo Neyman-Pearson test we develop is useful when one has a reliable model of faulty data, while the Fisher one is applicable when one may not have a model of faults, which may occur when dealing with third-party data, like GNSS data of transportation system users. These statistical tests can be combined with a particle filter to obtain a Monte Carlo state estimation scheme that is robust to faulty or outlier data. We present a synthetic freeway traffic state estimation problem where the filters are able to reject simulated faulty GNSS measurements. The fault-model-free Fisher filter, while underperforming the Neyman-Pearson one when the latter has an accurate fault model, outperforms it when the assumed fault model is incorrect.