Empirical Error Rate Research Articles

Bayesian Estimation of Allele-Specific Expression in the Presence of Phasing Uncertainty.

Allele-specific expression (ASE) analyses aim to detect imbalanced expression of maternal versus paternal copies of an autosomal gene. Such allelic imbalance can result from a variety of cis-acting causes, including disruptive mutations within one copy of a gene that impact the stability of transcripts, as well as regulatory variants outside the gene that impact transcription initiation. Current methods for ASE estimation suffer from a number of shortcomings, such as relying on only one variant within a gene, assuming perfect phasing information across multiple variants within a gene, or failing to account for alignment biases and possible genotyping errors. We developed BEASTIE, a Bayesian hierarchical model designed for precise ASE quantification at the gene level, based on given genotypes and RNA-Seq data. BEASTIE addresses the complexities of allelic mapping bias, genotyping error, and phasing errors by incorporating empirical phasing error rates derived from Genome-in-a-Bottle individual NA12878. BEASTIE surpasses existing methods in accuracy, especially in scenarios with high phasing errors. This improvement is critical for identifying rare genetic variants often obscured by such errors. Through rigorous validation on simulated data and application to real data from the 1000 Genomes Project, we establish the robustness of BEASTIE. These findings underscore the value of BEASTIE in revealing patterns of ASE across gene sets and pathways. The software is freely available from https://github.com/x811zou/BEASTIE . BEASTIE is available as Python source code and as a Docker image. Additional information is available online.

Homogeneity Test of Many-to-One Relative Risk Ratios in Unilateral and Bilateral Data with Multiple Groups

In medical clinical studies, we often encounter paired organs’ unilateral or bilateral data. For bilateral data, there exists an intraclass correlation between paired organs. Under an intraclass correlation model, this paper proposes asymptotic statistics for testing the equality of many-to-one relative risk ratios in combined unilateral and bilateral data. Furthermore, we calculate the explicit expressions of these statistics. Moreover, these procedures are adequate to solve the hypothesis problems of unilateral or bilateral data. Through comparison, the simulation results show that the score test has a robust empirical type-I error rate and sufficient power. We provide a clinical trial of acute otitis media to illustrate our proposed methods.

Well-calibrated confidence measures for multi-label text classification with a large number of labels

We extend our previous work on Inductive Conformal Prediction (ICP) for multi-label text classification and present a novel approach for addressing the computational inefficiency of the Label Powerset (LP) ICP, arrising when dealing with a high number of unique labels. We present experimental results using the original and the proposed efficient LP-ICP on two English and one Czech language data-sets. Specifically, we apply the LP-ICP on three deep Artificial Neural Network (ANN) classifiers of two types: one based on contextualised (bert) and two on non-contextualised (word2vec) word-embeddings. In the LP-ICP setting we assign nonconformity scores to label-sets from which the corresponding p-values and prediction-sets are determined. Our approach deals with the increased computational burden of LP by eliminating from consideration a significant number of label-sets that will surely have p-values below the specified significance level. This reduces dramatically the computational complexity of the approach while fully respecting the standard CP guarantees. Our experimental results show that the contextualised-based classifier surpasses the non-contextualised-based ones and obtains state-of-the-art performance for all data-sets examined. The good performance of the underlying classifiers is carried on to their ICP counterparts without any significant accuracy loss, but with the added benefits of ICP, i.e. the confidence information encapsulated in the prediction sets. We experimentally demonstrate that the resulting prediction sets can be tight enough to be practically useful even though the set of all possible label-sets contains more than 1e+16 combinations. Additionally, the empirical error rates of the obtained prediction-sets confirm that our outputs are well-calibrated.

Classification of Gaussian spatio-temporal data with stationary separable covariances

The novel approach to classification of spatio-temporal data based on Bayes discriminant functions is developed. We focus on the problem of supervised classifying of the spatiotemporal Gaussian random field (GRF) observation into one of two classes specified by different drift parameters, separable nonlinear covariance functions and nonstationary label field. The performance of proposed classification rule is validated by the values of local Bayes and empirical error rates realized by leave one out procedure. A simulation study for spatial covariance functions belonging to powered-exponential family and temporal covariance functions of AR(1) models is carried out. The influence of the values of spatial and temporal covariance parameters to error rates for several label field models are studied. The results showed that the proposed classification methodology can be applied successfully in practice with small error rates and can be a useful tool for discriminant analysis of spatio-temporal data.

Towards Continuous and Ambulatory Blood Pressure Monitoring: Methods for Efficient Data Acquisition for Pulse Transit Time Estimation.

We developed a prototype for measuring physiological data for pulse transit time (PTT) estimation that will be used for ambulatory blood pressure (BP) monitoring. The device is comprised of an embedded system with multimodal sensors that streams high-throughput data to a custom Android application. The primary focus of this paper is on the hardware–software codesign that we developed to address the challenges associated with reliably recording data over Bluetooth on a resource-constrained platform. In particular, we developed a lossless compression algorithm that is based on optimally selective Huffman coding and Huffman prefixed coding, which yields virtually identical compression ratios to the standard algorithm, but with a 67–99% reduction in the size of the compression tables. In addition, we developed a hybrid software–hardware flow control method to eliminate microcontroller (MCU) interrupt-latency related data loss when multi-byte packets are sent from the phone to the embedded system via a Bluetooth module at baud rates exceeding 115,200 bit/s. The empirical error rate obtained with the proposed method with the baud rate set to 460,800 bit/s was identically equal to . Our robust and computationally efficient physiological data acquisition system will enable field experiments that will drive the development of novel algorithms for PTT-based continuous BP monitoring.

Statistical Agnostic Mapping: A framework in neuroimaging based on concentration inequalities

In the 1970s a novel branch of statistics emerged focusing its effort on the selection of a function for the pattern recognition problem that would fulfill a relationship between the quality of the approximation and its complexity. This theory is mainly devoted to problems of estimating dependencies in the case of limited sample sizes, and comprise all the empirical out-of sample generalization approaches; e.g. cross validation (CV). In this paper a data-driven approach based on concentration inequalities is designed for testing competing hypothesis or comparing different models. In this sense we derive a Statistical Agnostic (non-parametric) Mapping (SAM) for neuroimages at voxel or regional levels which is able to: (i) relieve the problem of instability with limited sample sizes when estimating the actual risk via CV; and (ii) provide an alternative way of Family-wise-error (FWE) corrected p-value maps in inferential statistics for hypothesis testing. Using several neuroimaging datasets (containing large and small effects) and random task group analyses to compute empirical familywise error rates, this novel framework resulted in a model validation method for small samples over dimension ratios, and a less-conservative procedure than FWE p-value correction to determine the significance maps from the inferences made using small upper bounds of the actual risk.

A Test for Independence in High-Dimensional Normal Data

The hypothesis of complete independence is necessary for many statistical inferences. Classical testing procedures can not be applied to test this hypothesis in high-dimensional data. In this paper, a simple test statistic is presented for testing complete independence in multivariate high dimensional normal data. Using the theory of martingales, the asymptotic normality of the test statistic is established. In order to evaluate the performance of the proposed test and compare it with existing procedures, a simulation study was conducted. The simulation results indicate that the proposed test has an empirical type-I error rate with an average relative error less than the available tests. An application of the proposed method for gene expression clinical prostate data is presented.

Covert Device Association Among Colluding Apps via Edge Processor Workload

While thriving application (app) distribution systems involving incentivized third-party app vendors are desirable for the emerging edge computing paradigm, they also bring security challenges as faced by the current mobile app distribution systems. This article studies a threat called covert device association , in which the vendors of two apps collude to figure out which of their app installations run on the same edge device. The threat can widely spread when the two apps are popular. It is also a stepping stone for: 1) the de-anonymization attacks against the users anonymous to one of the two vendors and 2) privilege escalation in which the two colluding vendors have united privileges. We show that the threat can be implemented via a reliable and ubiquitous covert channel based on the edge device’s processor workload without requiring any privileged permissions. We present the implementation details for three attack scenarios of: 1) two Android apps; 2) an Android app and a Web session running in the mobile Tor browser; and 3) two Android Things apps. Evaluation on two smartphones and an embedded edge device shows that the covert channel gives at least 0.25 b/s data rate with zero empirical bit error rate and the covert device association can be completed within 3.2 min.

Secure Information Fusion using Local Posterior for Distributed Cyber-Physical Systems

In modern distributed cyber-physical systems (CPS), information fusion often plays a key role in automate and self-adaptive decision making process. However, given the heterogeneous and distributed nature of modern CPSs, it is a great challenge to operate CPSs with the compromised data integrity and unreliable communication links. In this paper, we study the distributed state estimation problem under the false data injection attack (FDIA) with probabilistic communication networks. We propose an integrated ”detection + fusion” solution, which is based on the Kullback-Leibler divergences (KLD) between local posteriors and therefore does not require the exchange of raw sensor data. For the FDIA detection step, the KLDs are used to cluster nodes in the probability space and to partition the space into secure and insecure subspaces. By approximating the distribution of the KLDs with a general <inline-formula><tex-math notation="LaTeX">$\chi ^2$</tex-math></inline-formula> distribution and calculating its tail probability, we provide an analysis of the detection error rate. For the information fusion step, we discuss the potential risk of double counting the shared prior information in the KLD-based consensus formulation method. We show that if the local posteriors are updated from the shared prior, the increased number of neighbouring nodes will lead to the diminished information gain. To overcome this problem, we propose a near-optimal distributed information fusion solution with properly weighted prior and data likelihood. Finally, we present simulation results for the integrated solution. We discuss the impact of network connectivity on the empirical detection error rate and the accuracy of state estimation.

Computational Dissociation of Dopaminergic and Cholinergic Effects on Action Selection and Inhibitory Control

BackgroundPatients with schizophrenia make more errors than healthy subjects in the antisaccade task. In this paradigm, participants are required to inhibit a reflexive saccade to a target and to select the correct action (a saccade in the opposite direction). While the precise origin of this deficit is not clear, it has been connected to aberrant dopaminergic and cholinergic neuromodulation. MethodsTo study the impact of dopamine and acetylcholine on inhibitory control and action selection, we administered two selective drugs (levodopa 200 mg/galantamine 8 mg) to healthy volunteers (N = 100) performing the antisaccade task. The computational model SERIA (stochastic early reaction, inhibition, and late action) was employed to separate the contribution of inhibitory control and action selection to empirical reaction times and error rates. ResultsModeling suggested that levodopa improved action selection (at the cost of increased reaction times) but did not have a significant effect on inhibitory control. By contrast, according to our model, galantamine affected inhibitory control in a dose-dependent fashion, reducing inhibition failures at low doses and increasing them at higher levels. These effects were sufficiently specific that the computational analysis allowed for identifying the drug administered to an individual with 70% accuracy. ConclusionsOur results do not support the hypothesis that elevated tonic dopamine strongly impairs inhibitory control. Rather, levodopa improved the ability to select correct actions. However, inhibitory control was modulated by cholinergic drugs. This approach may provide a starting point for future computational assays that differentiate neuromodulatory abnormalities in heterogeneous diseases like schizophrenia.

Evaluating Equivalence Testing Methods for Measurement Invariance

Measurement Invariance (MI) is often concluded from a nonsignificant chi-square difference test. Researchers have also proposed using change in goodness-of-fit indices (GOFs) instead. Both of these commonly used methods for testing MI have important limitations. To combat these issues, To combat these issues, it was proposed using an equivalence test (EQ) to replace the chi-square difference test commonly used to test MI. Due to concerns with the EQ's power, and adjusted version (EQ-A) was created, but provides little evaluation of either procedure. The current study evaluated the Type I error and power of both the EQ and EQ-A, and compared their performance to that of the traditional chi-square difference test and GOFs. The EQ was the only procedure that maintained empirical error rates below the nominal alpha level. Results also highlight that the EQ requires larger sample sizes than traditional difference-based approaches or using equivalence bounds based on larger than conventional RMSEA values (e.g., > .05) to ensure adequate power rates. We do not recommend the proposed adjustment (EQ-A) over the EQ.

Error rates in proficiency testing in Australia

ABSTRACTRecent criticism of forensic science has focused on the fundamental aspects of the science, including the lack of supporting empirical studies, validation, accreditation, limitations and error rates. Proficiency tests are an essential component of accreditation and can be used to evaluate laboratory performance and identify systematic issues within components of the service provision. In 2016 we reported on the results of an analysis of 3176 CTS proficiency tests undertaken between 2005 and 2015 by Australian government service providers. The data analysed represented 43 unique CTS test types and covered 21 disciplines. Here we present further data from 2016 to 2017 and compare these results with those obtained from the previous study. These combined results further demonstrate that errors exist even though practitioners know they are examining proficiency tests and the tests undergo a review process. This study illustrates the need to continue to monitor trends in proficiency test results and also further highlights the need for well-designed, relevant, blind error rate studies to determine the approximate error rates for casework.

Cluster-level statistical inference in fMRI datasets: The unexpected behavior of random fields in high dimensions

Identifying regional effects of interest in MRI datasets usually entails testing a priori hypotheses across many thousands of brain voxels, requiring control for false positive findings in these multiple hypotheses testing. Recent studies have suggested that parametric statistical methods may have incorrectly modeled functional MRI data, thereby leading to higher false positive rates than their nominal rates. Nonparametric methods for statistical inference when conducting multiple statistical tests, in contrast, are thought to produce false positives at the nominal rate, which has thus led to the suggestion that previously reported studies should reanalyze their fMRI data using nonparametric tools.To understand better why parametric methods may yield excessive false positives, we assessed their performance when applied both to simulated datasets of 1D, 2D, and 3D Gaussian Random Fields (GRFs) and to 710 real-world, resting-state fMRI datasets. We showed that both the simulated 2D and 3D GRFs and the real-world data contain a small percentage (<6%) of very large clusters (on average 60 times larger than the average cluster size), which were not present in 1D GRFs. These unexpectedly large clusters were deemed statistically significant using parametric methods, leading to empirical familywise error rates (FWERs) as high as 65%: the high empirical FWERs were not a consequence of parametric methods failing to model spatial smoothness accurately, but rather of these very large clusters that are inherently present in smooth, high-dimensional random fields. In fact, when discounting these very large clusters, the empirical FWER for parametric methods was 3.24%. Furthermore, even an empirical FWER of 65% would yield on average less than one of those very large clusters in each brain-wide analysis. Nonparametric methods, in contrast, estimated distributions from those large clusters, and therefore, by construct rejected the large clusters as false positives at the nominal FWERs. Those rejected clusters were outlying values in the distribution of cluster size but cannot be distinguished from true positive findings without further analyses, including assessing whether fMRI signal in those regions correlates with other clinical, behavioral, or cognitive measures. Rejecting the large clusters, however, significantly reduced the statistical power of nonparametric methods in detecting true findings compared with parametric methods, which would have detected most true findings that are essential for making valid biological inferences in MRI data. Parametric analyses, in contrast, detected most true findings while generating relatively few false positives: on average, less than one of those very large clusters would be deemed a true finding in each brain-wide analysis. We therefore recommend the continued use of parametric methods that model nonstationary smoothness for cluster-level, familywise control of false positives, particularly when using a Cluster Defining Threshold of 2.5 or higher, and subsequently assessing rigorously the biological plausibility of the findings, even for large clusters. Finally, because nonparametric methods yielded a large reduction in statistical power to detect true positive findings, we conclude that the modest reduction in false positive findings that nonparametric analyses afford does not warrant a re-analysis of previously published fMRI studies using nonparametric techniques.

Class of novel broadband chaos‐based coherent communication systems

As opposed to chaotic communication systems where a message signal directly modulates a broadband chaotic signal, it is also possible to indirectly utilise a chaotic signal to form a chaos-based communication system. This study proposes a class of novel broadband chaos-based coherent communication systems and demonstrates that their theoretical probability of error expressions match those of their counterpart conventional narrowband systems in the additive white Gaussian noise and the Rayleigh flat fading channel. The results are confirmed with the corresponding empirical bit error rate (BER) simulations. It is demonstrated that the chaos-based quadrature phase shift keying system offers an overall best performance while the chaos-based amplitude shift keying and frequency shift keying equivalent systems offer a possibility of performance optimisation. Most importantly, it is shown that the proposed systems outperform the chaotic communication systems in terms of BER while also being more practicable. The proposed systems thus hold high promise as the chaotic communication systems pose implementation difficulties, lack robust synchronisation techniques and are generally considered of academic interest only.

The Analysis of Australian Proficiency Test Data over a Ten-Year Period

ABSTRACTRecent criticism of forensic science has focused on the fundamental aspects of the science, including supporting empirical studies, validation, accreditation, limitations and error rates. P...

Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates

The most widely used task functional magnetic resonance imaging (fMRI) analyses use parametric statistical methods that depend on a variety of assumptions. In this work, we use real resting-state data and a total of 3 million random task group analyses to compute empirical familywise error rates for the fMRI software packages SPM, FSL, and AFNI, as well as a nonparametric permutation method. For a nominal familywise error rate of 5%, the parametric statistical methods are shown to be conservative for voxelwise inference and invalid for clusterwise inference. Our results suggest that the principal cause of the invalid cluster inferences is spatial autocorrelation functions that do not follow the assumed Gaussian shape. By comparison, the nonparametric permutation test is found to produce nominal results for voxelwise as well as clusterwise inference. These findings speak to the need of validating the statistical methods being used in the field of neuroimaging.

Empirical estimation of sequencing error rates using smoothing splines.

BackgroundNext-generation sequencing has been used by investigators to address a diverse range of biological problems through, for example, polymorphism and mutation discovery and microRNA profiling. However, compared to conventional sequencing, the error rates for next-generation sequencing are often higher, which impacts the downstream genomic analysis. Recently, Wang et al. (BMC Bioinformatics 13:185, 2012) proposed a shadow regression approach to estimate the error rates for next-generation sequencing data based on the assumption of a linear relationship between the number of reads sequenced and the number of reads containing errors (denoted as shadows). However, this linear read-shadow relationship may not be appropriate for all types of sequence data. Therefore, it is necessary to estimate the error rates in a more reliable way without assuming linearity. We proposed an empirical error rate estimation approach that employs cubic and robust smoothing splines to model the relationship between the number of reads sequenced and the number of shadows.ResultsWe performed simulation studies using a frequency-based approach to generate the read and shadow counts directly, which can mimic the real sequence counts data structure. Using simulation, we investigated the performance of the proposed approach and compared it to that of shadow linear regression. The proposed approach provided more accurate error rate estimations than the shadow linear regression approach for all the scenarios tested. We also applied the proposed approach to assess the error rates for the sequence data from the MicroArray Quality Control project, a mutation screening study, the Encyclopedia of DNA Elements project, and bacteriophage PhiX DNA samples.ConclusionsThe proposed empirical error rate estimation approach does not assume a linear relationship between the error-free read and shadow counts and provides more accurate estimations of error rates for next-generation, short-read sequencing data.Electronic supplementary materialThe online version of this article (doi:10.1186/s12859-016-1052-3) contains supplementary material, which is available to authorized users.

Reliable Robust Regression Diagnostics

SummaryMotivated by the requirement of controlling the number of false discoveries that arises in several application fields, we study the behaviour of diagnostic procedures obtained from popular high‐breakdown regression estimators when no outlier is present in the data. We find that the empirical error rates for many of the available techniques are surprisingly far from the prescribed nominal level. Therefore, we propose a simulation‐based approach to correct the liberal diagnostics and reach reliable inferences. We provide evidence that our approach performs well in a wide range of settings of practical interest and for a variety of robust regression techniques, thus showing general appeal. We also evaluate the loss of power that can be expected from our corrections under different contamination schemes and show that this loss is often not dramatic. Finally, we detail some possible extensions that may further enhance the applicability of the method.

Chaos‐based BPSK communication system

A novel broadband chaos-based coherent binary phase shift keying (BPSK) communication system is proposed and it is demonstrated that its theoretical probability of error expression matches that of a conventional narrowband BPSK system. The result is confirmed with the corresponding empirical bit error rate (BER) simulation. Most importantly, it is shown that the proposed system outperforms the chaotic communication systems in terms of BER while also being more practicable. The proposed system thus holds high promise as the chaotic communication systems pose implementation difficulties, lack robust synchronisation techniques and are generally considered of academic interest only.

Definition of loss functions for learning from imbalanced data to minimize evaluation metrics.

Most learning algorithms for classification use objective functions based on regularized and/or continuous versions of the 0-1 loss function. Moreover, the performance of the classification models is usually measured by means of the empirical error or misclassification rate. Nevertheless, neither those loss functions nor the empirical error is adequate for learning from imbalanced data. In these problems, the empirical error is uninformative about the performance of the classifier and the loss functions usually produce models that are shifted to the majority class. This study defines the loss function L BER whose associated empirical risk is equal to the BER. Our results show that classifiers based on our L BER loss function are optimal in terms of the BER evaluation metric. Furthermore, the boundaries of the classifiers were invariant to the imbalance ratio of the training dataset. The L BER-based models outperformed the 0-1-based models and other algorithms for imbalanced data in terms of BER, regardless of the prevalence of the positive class. Finally, we demonstrate the equivalence of the loss function to the method of inverted prior probabilities, and we define the family of loss functions L WER that is associated with any WER evaluation metric by the generalization of L BER.