Error Rate Estimation Research Articles

Multiple comparison correction methods for whole-body magnetic resonance imaging.

.Purpose: Voxel-level hypothesis testing on images suffers from test multiplicity. Numerous correction methods exist, mainly applied and evaluated on neuroimaging and synthetic datasets. However, newly developed approaches like Imiomics, using different data and less common analysis types, also require multiplicity correction for more reliable inference. To handle the multiple comparisons in Imiomics, we aim to evaluate correction methods on whole-body MRI and correlation analyses, and to develop techniques specifically suited for the given analyses.Approach: We evaluate the most common familywise error rate (FWER) limiting procedures on whole-body correlation analyses via standard (synthetic no-activation) nominal error rate estimation as well as smaller prior-knowledge based stringency analysis. Their performance is compared to our anatomy-based method extensions.Results: Results show that nonparametric methods behave better for the given analyses. The proposed prior-knowledge based evaluation shows that the devised extensions including anatomical priors can achieve the same power while keeping the FWER closer to the desired rate.Conclusions: Permutation-based approaches perform adequately and can be used within Imiomics. They can be improved by including information on image structure. We expect such method extensions to become even more relevant with new applications and larger datasets.

NIST Ballistics Toolmark Research Database.

In 2009, a report by the National Academies called into question, amongst other issues, the objectivity of visual toolmark identification by firearms examiners. The National Academies recommended development of objective toolmark identification criteria and error rate estimates. Industry, academia, and government laboratories are pursuing two promising approaches towards this goal: 1) development of mathematical criteria and advanced algorithms for the objective and automated identification and scoring of potential matches, and 2) supplementing traditional reflectance microscopy images with three- dimensional surface topography measurement data. Development and validation of both these approaches to objective toolmark identification are hindered by a lack of access to toolmark data sets that 1) represent the large variety of ballistic toolmarks encountered by toolmark examiners, and 2) represent challenging identification scenarios, such as those posed by consecutively manufactured firearms components. The NIST Ballistics Toolmark Research Database enables researchers to test and validate new approaches to objective, mathematics-based, toolmark identification while easing the transition to three- dimensional surface topography data. The database will provide a foundation for a scientific knowledge base on the degree of similarity that can be found between marks made by different firearms and the variability in marks made by an individual firearm. The current, fairly limited, knowledge base is a fundamental barrier to the development and validation of objective mathematical similarity criteria, and associated confidence limits, applicable to a broad range of firearms and ammunition brands.

Bit Error Rate Evaluation in Atmospheric Blustery for Slant Path Spread of Super Lorentz Gaussian Beam

The formula of the scintillation index fixated on bit error rate of Super Lorentz Gaussian beam in turbulent atmosphere is derivative by using the Huygens Fresnel Process to evaluate the average bit error rate. The scintillation index guides of a higher beam order engrossed Super Lorentz Gaussian beam flat and Cartesian coordinates are compared. The average signal-to-noise ratio is computed via the versus scintillation index and bit error rate by using the formula of log-normal dispersed intensity for the beam order and absorbed the Super Lorentz Gaussian beams (SLG) is propagated in the slant path, which display several factors such as source size, focal length and wavelength, for example, 0.8 and 1.55 μm. Absorbed beams are initiated to take significant compensations over higher order beams. According to the two desired wavelength distributions processed theoretically, the results of bit error rate estimation were started to the value of source size rises, the system is improved and scintillation index, bit error rate decreased, the quality of bit error rate was enhanced, and the elevation was decreased. Bit error rate quality estimation of full reference objective relay on the feedback by SLG22 beam has the better bit error rate.

Cyclostationary Characterization of Radiated Emissions in Digital Electronic Devices

Stochastic electromagnetic fields originating from processes in digital circuitry affect the operation of the circuit, increase the error rate in the transmission of digital signals, degrade the circuit's performance, yield malfunction of the system or even lead to a complete failure of the system's operation. Interference from stochastic fields are not only disruptive but are becoming more common due to the increasing complexity of modern electronics. Stochastic electromagnetic fields under common circumstances are due to a mixture of emissions from different sources, which makes a quantification of stochastic fields very challenging. Since digital signals are clocked, the ensemble averages of the electromagnetic field transients representing the digital signal exhibit a periodic time dependence with the period of the clock interval. The probabilistic model of the cyclostationary random process was composed by appropriate transformations of the initial discrete random process of the binary symbols. Exploiting the cyclostationary characteristics of stochastic fields greatly simplifies their analysis. The physical signal propagating along the transmission line is considered also a source of the radiating electromagnetic field which preserves the cyclostationary properties of the binary sequence. The stochastic process of the measured near-field emissions inherits probability characteristicsfrom the random process of the physical signal in the transmission line. This paper presents an approach for quantifying emissions and coupling of stochastic fields using their cyclostationary properties. For practical evaluation of cyclostationary characteristics the statistical cyclic averaging of the measured signals are used. The important feature of the statistical cyclic averaging procedure is its capability to distinguish cyclostationary signals from a mixture of stochastic processes such as stationary noise with deterministic signals and even from other cyclostationary signals with distinct cyclic frequencies. The measurement setup for observing electromagnetic emissions from printed circuit boards (PCBs) hosting digital electronic devices comprises an automated positioning system with a scanning near-field probe attached to the moving scanner head. This scanning near-field probe and a properly position fixed or also movable reference probe are connected to the inputs of a multi-channel digital oscilloscope for simultaneous and synchronous sampling and registering of the magnetic field components. The registered data from the oscilloscope was transferred to a data acquisition computer. After the postprocessing procedure, the spatial localization of distributed radiating sources on the surface of the PCB was performed in conjunction with determination of their average power characteristics and specified cyclic frequencies. The proposed cyclic averaging was applied to the spatial separation of radiating sources with different bit rates and for the quantitative characterization of crosstalk and coupling between transmission lines carrying the data sequencesand subsequent estimation of bit error rate (BER) caused by the crosstalk using their cyclostationary properties.

Joint Random Subcarrier Selection and Channel-Based Artificial Signal Design Aided PLS

A novel method for providing physical layer security (PLS) depending on the randomness of wireless channel is proposed. Specifically, a channel-based joint random subcarrier selection and artificial signal design are introduced to protect the communication in the presence of a strong passive eavesdropper. Our analysis assumes a window-based subcarrier selection method in which the strongest subcarriers in each window are selected. Chosen subcarriers are considered for secret sequence extraction. In addition the generated channel dependent secret sequence is used for both random subcarrier selection and artificial signal design. We evaluate the efficiency of the proposed method through some representative metrics, such as secret sequence disagreement rate (SSDR), throughput and bit error rate (BER), in both perfect and imperfect channel estimation cases. Simulation results are presented and insightful discussions are drawn.

The effect of the terminal penalty in Receding Horizon Control for a class of stabilization problems

The Receding Horizon Control (RHC) strategy consists in replacing an infinite-horizon stabilization problem by a sequence of finite-horizon optimal control problems, which are numerically more tractable. The dynamic programming principle ensures that if the finite-horizon problems are formulated with the exact value function as a terminal penalty function, then the RHC method generates an optimal control. This article deals with the case where the terminal cost function is chosen as a cut-off Taylor approximation of the value function. The main result is an error rate estimate for the control generated by such a method, when compared with the optimal control. The obtained estimate is of the same order as the employed Taylor approximation and decreases at an exponential rate with respect to the prediction horizon. To illustrate the methodology, the article focuses on a class of bilinear optimal control problems in infinite-dimensional Hilbert spaces.

Model Evaluation for Forecasting Traffic Accident Severity in Rainy Seasons Using Machine Learning Algorithms: Seoul City Study

There have been numerous studies on traffic accidents and their severity, particularly in relation to weather conditions and road geometry. In these studies, traditional statistical methods have been employed, such as linear regression, logistic regression, and negative binomial regression modeling, which are the most common linear and non-linear regression analysis methods. In this research, machine learning architecture was applied to this problem using the random forest, artificial neural network, and decision tree techniques to ascertain the strengths and weaknesses of these methods. Three data sets were used: road geometry data, precipitation data, and traffic accident data over nine years corresponding to the Naebu Expressway, which is located in Seoul, Korea. For the model evaluation, three measures were employed: the out-of-bag estimate of error rate (OOB), mean square error (MSE), and root mean square error (RMSE). The low mean OOB, MSE, and RMSE observed in the results obtained using the proposed random forest model demonstrate its accuracy.

M-ary phase-shift keying-based single-input-multiple-output free space optical communication system with pointing errors over a gamma-gamma fading channel.

The performance of free space optical (FSO) communication systems is severely affected due to a number of reasons, such as atmospheric impairments, fading due to turbulence, and misalignment between transmitter and receiver sections. In this paper, we have analyzed the atmospheric turbulence and pointing error effect on M-ary phase-shift keying modulation (M-PSK)-based FSO system link performance over a gamma-gamma fading channel. In our analysis, the effect of a weak-to-strong atmospheric turbulence channel condition and pointing errors are taken into consideration. To deal with the performance degradation, a single-input-multiple-output (SIMO) link along with maximal ratio combining (MRC) diversity scheme is employed. The closed-form expressions for the estimation of average bit error rate (BER), outage probability, and spectral efficiency are derived in terms of the Meijer G-function. The analytical results are presented to indicate the effect of turbulence and pointing errors on BER. The outage probability and spectral efficiency of the proposed SIMO-FSO system with MRC technique improved significantly. The accuracy of all the analytical results is verified by Monte Carlo simulation.

Antibiotic resistance prediction for Mycobacterium tuberculosis from genome sequence data with Mykrobe

Two billion people are infected with Mycobacterium tuberculosis, leading to 10 million new cases of active tuberculosis and 1.5 million deaths annually. Universal access to drug susceptibility testing (DST) has become a World Health Organization priority. We previously developed a software tool, Mykrobe predictor, which provided offline species identification and drug resistance predictions for M. tuberculosis from whole genome sequencing (WGS) data. Performance was insufficient to support the use of WGS as an alternative to conventional phenotype-based DST, due to mutation catalogue limitations. Here we present a new tool, Mykrobe, which provides the same functionality based on a new software implementation. Improvements include i) an updated mutation catalogue giving greater sensitivity to detect pyrazinamide resistance, ii) support for user-defined resistance catalogues, iii) improved identification of non-tuberculous mycobacterial species, and iv) an updated statistical model for Oxford Nanopore Technologies sequencing data. Mykrobe is released under MIT license at https://github.com/mykrobe-tools/mykrobe. We incorporate mutation catalogues from the CRyPTIC consortium et al. (2018) and from Walker et al. (2015), and make improvements based on performance on an initial set of 3206 and an independent set of 5845 M. tuberculosis Illumina sequences. To give estimates of error rates, we use a prospectively collected dataset of 4362 M. tuberculosis isolates. Using culture based DST as the reference, we estimate Mykrobe to be 100%, 95%, 82%, 99% sensitive and 99%, 100%, 99%, 99% specific for rifampicin, isoniazid, pyrazinamide and ethambutol resistance prediction respectively. We benchmark against four other tools on 10207 (=5845+4362) samples, and also show that Mykrobe gives concordant results with nanopore data. We measure the ability of Mykrobe-based DST to guide personalized therapeutic regimen design in the context of complex drug susceptibility profiles, showing 94% concordance of implied regimen with that driven by phenotypic DST, higher than all other benchmarked tools.

Estimation of human error rate in underground coal mines through retrospective analysis of mining accident reports and some error reduction strategies

Human error is one of the leading causes of workplace accidents and a pressing threat to system safety. The management and control of human errors need their identification in the system activities and prioritisation of errors. Knowledge of error rates or probabilities is essential for their prioritisation and is a prerequisite for designing control measures of human error to enhance system safety. Traditional approaches estimate the human error rate based on experts’ opinion, which suffers from the inherent uncertainty of human judgments. This research proposes a methodology for estimation of human error rate from a retrospective analysis of accident reports using fuzzy mathematical concepts. The proposed approach uses accident reports of underground coal mines for assessing the human error rates of essential mining activities, identifying the critical activities and error types. It also suggests some error reduction strategies for devising an intervention to accidents.

Antibiotic resistance prediction for Mycobacterium tuberculosis from genome sequence data with Mykrobe.

Two billion people are infected with Mycobacterium tuberculosis, leading to 10 million new cases of active tuberculosis and 1.5 million deaths annually. Universal access to drug susceptibility testing (DST) has become a World Health Organization priority. We previously developed a software tool, Mykrobe predictor, which provided offline species identification and drug resistance predictions for M. tuberculosis from whole genome sequencing (WGS) data. Performance was insufficient to support the use of WGS as an alternative to conventional phenotype-based DST, due to mutation catalogue limitations. Here we present a new tool, Mykrobe, which provides the same functionality based on a new software implementation. Improvements include i) an updated mutation catalogue giving greater sensitivity to detect pyrazinamide resistance, ii) support for user-defined resistance catalogues, iii) improved identification of non-tuberculous mycobacterial species, and iv) an updated statistical model for Oxford Nanopore Technologies sequencing data. Mykrobe is released under MIT license at https://github.com/mykrobe-tools/mykrobe. We incorporate mutation catalogues from the CRyPTIC consortium et al. (2018) and from Walker et al. (2015), and make improvements based on performance on an initial set of 3206 and an independent set of 5845 M. tuberculosis Illumina sequences. To give estimates of error rates, we use a prospectively collected dataset of 4362 M. tuberculosis isolates. Using culture based DST as the reference, we estimate Mykrobe to be 100%, 95%, 82%, 99% sensitive and 99%, 100%, 99%, 99% specific for rifampicin, isoniazid, pyrazinamide and ethambutol resistance prediction respectively. We benchmark against four other tools on 10207 (=5845+4362) samples, and also show that Mykrobe gives concordant results with nanopore data. We measure the ability of Mykrobe-based DST to guide personalized therapeutic regimen design in the context of complex drug susceptibility profiles, showing 94% concordance of implied regimen with that driven by phenotypic DST, higher than all other benchmarked tools.

Accurate BER Estimation Scheme Based on K-Means Clustering Assisted Gaussian Approach for Arbitrary Modulation Format

We propose a novel bit error rate (BER) estimation scheme based on a k -means clustering algorithm assisted Gaussian approach (GA). This method can implement accurate BER estimation within a short symbol sequence and is applicable for arbitrary modulation formats. First, after carrier phase recovery (CPR), the k -means clustering algorithm is used to partition clusters and find the respective centroids of each cluster. The means and variances are calculated from the symbols of each cluster. Compared with the decision-directed method based on power normalization, our method can find more precise means and further obtain more accurate variances. Subsequently, with the accurate means and variances, the resultant probability density function (PDF) of each symbol under Gaussian assumption is integrated over the respective decision zone to calculate symbol error rate (SER). Finally, the general conversion factor from SER to BER is introduced by taking into account the coding information of adjacent symbols. Therefore, the accurate BER estimation is attributed to the more accurate statistical parameter calculation and general SER-to-BER conversion schemes. The proposed scheme is verified in 34 GBaud polarization division multiplexing (PDM)-QPSK/8-QAM/16-QAM experiments. Compared with error vector magnitude (EVM)-to-BER conversion and GA+ common approximation (CA) scheme, the better estimation accuracy in the BER range from 10−6–10−2 is achieved successfully with only 10000 symbols. More specifically, our method has a significant improvement in estimation accuracy for non-Gray-mapped signals under low optical signal-to-noise ratio (OSNR). The BER estimation error can be reduced from 60 to 14% with PDM-8-QAM signal considered when actual BER is around 10−3.

Fired bullet signature correlation using the Congruent Matching Profile Segments (CMPS) method

We introduce the Congruent Matching Profile Segments (CMPS) method for objective comparison of striated tool marks and apply it to bullet signature correlations. The method is derived from the congruent matching cell (CMC) method developed for the comparison of impressed tool marks. The proposed method is designed to increase comparison accuracy by addressing the comparison challenges caused by striae profiles with different lateral scales, varying vertical (height) scales, and sections that are poorly marked or have little to no similarity. Instead of correlating the entire profiles extracted from striated tool marks, the method divides one of the compared profiles into segments. Each segment is then correlated with the other profile. The CMPS method uses the normalized cross-correlation function with multiple correlation peak inspection to determine the number of profile segments that have both significant topography similarity and a congruent registration position. Initial tests were performed on the land engraved areas (LEAs) of 35 bullets fired from 10 consecutively manufactured pistol barrels. The results show clear separation between the CMPS scores of the 549 known non-matching (KNM) LEA profiles and the 46 known matching (KM) LEA profiles. These results are an improvement over those obtained using the correlation coefficient score of whole profiles. The large number of CMPS segment correlations may facilitate a statistical approach to error rate estimations.

Perceptions and estimates of error rates in forensic science: A survey of forensic analysts

Every scientific technique features some error, and legal standards for the admissibility of scientific evidence (e.g., Daubert v. Merrill Dow Pharmaceuticals, Inc., 1993; Kumho Tire Co v. Carmichael, 1999) guide trial courts to consider known error rates. However, recent reviews of forensic science conclude that error rates for some common techniques are not well-documented or even established (e.g., NAS, 2009; PCAST, 2016). Furthermore, many forensic analysts have historically denied the presence of error in their field. Therefore, it is important to establish what forensic scientists actually know or believe about errors rates in their disciplines. We surveyed 183 practicing forensic analysts to examine what they think and estimate about error rates in their various disciplines. Results revealed that analysts perceive all types of errors to be rare, with false positive errors even more rare than false negatives. Likewise, analysts typically reported that they prefer to minimize the risk of false positives over false negatives. Most analysts could not specify where error rates for their discipline were documented or published. Their estimates of error in their fields were widely divergent – with some estimates unrealistically low.

Forensic science evidence: Naive estimates of false positive error rates and reliability

We do not know how often false positive reports are made in a range of forensic science disciplines. In the absence of this information it is important to understand the naive beliefs held by potential jurors about forensic science evidence reliability. It is these beliefs that will shape evaluations at trial. This descriptive study adds to our knowledge about naive beliefs by: (1) measuring jury-eligible (lay) perceptions of reliability for the largest range of forensic science disciplines to date, over three waves of data collection between 2011 and 2016 (n=674); (2) calibrating reliability ratings with false positive report estimates; and (3) comparing lay reliability estimates with those of an opportunity sample of forensic practitioners (n=53). Overall the data suggest that both jury-eligible participants and practitioners consider forensic evidence highly reliable. When compared to best or plausible estimates of reliability and error in the forensic sciences these views appear to overestimate reliability and underestimate the frequency of false positive errors. This result highlights the importance of collecting and disseminating empirically derived estimates of false positive error rates to ensure that practitioners and potential jurors have a realistic impression of the value of forensic science evidence.

Smartwatch PPG Peak Detection Method for Sinus Rhythm and Cardiac Arrhythmia.

The aim of our work herein was to design a photoplethysmographic (PPG) peak detection algorithm which automatically detect and discriminate various cardiac rhythms-normal sinus rhythms (NSR), premature atrial contraction (PAC), premature ventricle contraction (PVC), and atrial fibrillation (AF)-for PPG signals collected on smartwatch. Compared with peak detection algorithm designed for NSR, the novelty is that our proposed peak detection algorithm can accurately estimate heart rates (HR) among various arrhythmias, which enhances the accuracy of AF screening. Our peak detection method is composed of a sequential series of algorithms that are combined to discriminate various arrhythmias, as described above. Moreover, a novel Poincaré plot scheme is used to discriminate AF with Rapid Ventricular Response (RVR) from normal basal heart rate AF. Moreover, the method is also able to differentiate PAC/PVC from NSR and AF. Our results show that the proposed peak detection algorithm provides significantly lower average beat-to-beat estimation error (> 40% lower) and mean heart rate estimation error (> 50% lower) when compared to a traditional peak detection algorithm that is known to be accurate for NSR. Our new approach allows more accurate HR estimation as it can account for various arrhythmias which previous PPG peak detection algorithms were designed solely for NSR.

Statistical analysis of randomized benchmarking

Randomized benchmarking and variants thereof, which we collectively call RB+, are widely used to characterize the performance of quantum computers because they are simple, scalable, and robust to state-preparation and measurement errors. However, experimental implementations of RB+ allocate resources suboptimally and make ad-hoc assumptions that undermine the reliability of the data analysis. In this paper, we propose a simple modification of RB+ which rigorously eliminates a nuisance parameter and simplifies the experimental design. We then show that, with this modification and specific experimental choices, RB+ efficiently provides estimates of error rates with multiplicative precision. Finally, we provide a simplified rigorous method for obtaining credible regions for parameters of interest and a heuristic approximation for these intervals that performs well in currently relevant regimes.

How Linkage Error Affects Hidden Markov Model Estimates: A Sensitivity Analysis

Abstract Hidden Markov models (HMMs) are increasingly used to estimate and correct for classification error in categorical, longitudinal data, without the need for a “gold standard,” error-free data source. To accomplish this, HMMs require multiple observations over time on a single indicator and assume that the errors in these indicators are conditionally independent. Unfortunately, this “local independence” assumption is often unrealistic, untestable, and a source of serious bias. Linking independent data sources can solve this problem by making the local independence assumption plausible across sources, while potentially allowing for local dependence within sources. However, record linkage introduces a new problem: the records may be erroneously linked or incorrectly not linked. In this paper, we investigate the effects of linkage error on HMM estimates of transitions between employment contract types. Our data come from linking a labor force survey to administrative employer records; this linkage yields two indicators per time point that are plausibly conditionally independent. Our results indicate that both false-negative and false-positive linkage error turn out to be problematic primarily if the error is large and highly correlated with the dependent variable. Moreover, under certain conditions, false-positive linkage error (mislinkage) in fact acts as another source of misclassification that the HMM can absorb into its error-rate estimates, leaving the latent transition estimates unbiased. In these cases, measurement error modeling already accounts for linkage error. Our results also indicate where these conditions break down and more complex methods would be needed.

Progressive Latent Models for Self-Learning Scene-Specific Pedestrian Detectors

The performance of offline learned pedestrian detectors significantly drops when they are applied to video scenes of various camera views, occlusions, and background structures. Learning a detector for each video scene can avoid the performance drop but it requires repetitive human effort on data annotation. In this paper, a self-learning approach is proposed, toward specifying a pedestrian detector for each video scene without any human annotation involved. Object locations in video frames are treated as latent variables and a progressive latent model (PLM) is proposed to solve such latent variables. The PLM is deployed as components of object discovery, object enforcement, and label propagation, which are used to learn the object locations in a progressive manner. With the difference of convex (DC) objective functions, PLM is optimized by a concave-convex programming algorithm. With specified network branches and loss functions, PLM is integrated with deep feature learning and optimized in an end-to-end manner. From the perspectives of convex regularization and error rate estimation, detailed optimization analysis and learning stability analysis of the proposed PLM are provided. The extensive experiments demonstrate that even without annotation involved the proposed self-learning approach outperforms weakly supervised learning approaches, while achieving comparable performance with transfer learning approaches.

Multi-matrix error estimation and reconciliation for quantum key distribution.

Post-processing is a significant step in quantum key distribution (QKD), which is used for correcting the quantum-channel noise errors and distilling identical corrected keys between two distant legitimate parties. Efficient error reconciliation protocol, which can lead to an increase in the secure key generation rate, is one of the main performance indicators of QKD setups. In this paper, we propose a multi-low-density parity-check codes based reconciliation scheme, which can provide remarkable perspectives for highly efficient information reconciliation. By testing our approach through data simulation, we show that the proposed scheme combining multi-syndrome-based error rate estimation allows for a more accurate estimation of the error rate as compared with random sampling and single-syndrome estimation techniques before the error correction, as well as a significant increase in the efficiency of the procedure without compromising security and sacrificing reconciliation efficiency.