False Negative Error Rates Research Articles

Optic nerve head drusen: a comparative study of 10 MHz and 20 MHz ultrasound probes

The aim of this study was to evaluate and compare 10-MHz and 20-MHz ultrasonography in the assessment of patients with optic nerve head drusen (ONHD). The design of the study was prospective, comparative and cross-sectional. Ultrasonographic examination with a 10 and 20MHz probe was performed in 45 eyes with suspected ONHD. The 20MHz probe showed drusen in 43 eyes (95.5%), while the 10MHz probe revealed drusen in only 33 eyes (73.3%, p=0.0001). The 10MHz probe showed surface drusen in 10 eyes (22.2%), while the 20MHz probe showed surface drusen in 14 eyes (31.1%) (sensitivity 71.4%; 95% CI [47.6-95.1%]). The 10MHz probe showed buried drusen in 23 eyes (23.1%), while the 20MHz probe showed buried drusen in 29 eyes (64.4%) (sensitivity 79.3%; 95% CI [56.6-86.2%]). The sensitivity was 76.7% with 10MHz probe compared to a 20MHz as gold standard. The use of 20MHz probe increased the sensitivity of buried disc drusen by 1.5 times and surface disc drusen by nearly 2 times. Using the 10MHz probe alone the false negative error rate was 83.3%. The 20MHz probe has shown itself to be an excellent method for the diagnosis of ONHD; it is more sensitive and reliable than 10MHz probe and should be considered in the management of patients with clinical evidence of ONHD.

Designing Pooling Systems for Noisy High-Throughput Protein-Protein Interaction Experiments Using Boolean Compressed Sensing

Group testing, also known as pooling, is a common technique used in high-throughput experiments in molecular biology to significantly reduce the number of tests required to identify rare biological interactions while correcting for experimental noise. Central to the group testing problem are 1) a pooling design that lays out how items are grouped together into pools for testing and 2) a decoder that interprets the results of the tested pools, identifying the active compounds. In this work, we take advantage of decoder guarantees from the field of compressed sensing (CS) to address the problem of efficient and reliable detection of biological interaction in noisy high-throughput experiments. We also use efficient combinatorial algorithms from group testing as well as established measurement matrices from CS to create pooling designs. First, we formulate the group testing problem in terms of a Boolean CS framework. We then propose a low-complexity l1-norm decoder to interpret pooling test results and identify active compounds. We demonstrate the robustness of the proposed l1-norm decoder in simulated experiments with false-positive and false-negative error rates typical of high-throughput experiments. When benchmarked against the current state-of-the-art methods, the proposed l1-norm decoder provides superior error correction for the majority of the cases considered while being notably faster computationally. Additionally, we test the performance of the l1-norm decoder against a real experimental data set, where 12,675 prey proteins were screened against 12 bait proteins. Lastly, we study the impact of different sparse pooling design matrices on decoder performance and show that the shifted transversal design (STD) is the most suitable among the pooling designs surveyed for biological applications of CS.

Evaluating ensemble forecasts of plant species distributions under climate change

Species distributions models (SDMs) are commonly used to assess potential species’ range shifts or extinction risk under climate change. It has been suggested that the use of ensemble forecasts, where a variety of model algorithms are used to generate consensus predictions, are preferred to individual SDMs by avoiding bias or prediction error inherent in a single modeling approach. Whereas several studies have assessed the performance of ensemble predictions using cross-validation or data-partitioning approaches, few studies have assessed the predictive accuracy of ensemble forecasts under climate change by using temporally independent model validation data. We used five SDM approaches to develop consensus forecasts of distributions of 145 vascular plant species from California in the 1930s and tested their projections against current distributions, a span of approximately 75 years. When evaluated with a portion of the model training data, consensus forecasts were highly accurate with an average AUC value of 0.97. False positive and false negative error rates were also low, exhibiting similar performance to random forest models. However, when evaluated with temporally independent data, the accuracy of consensus forecasts was similar to that of generalized linear and generalized additive models, with an average AUC value of 0.83. Our results suggest that the high levels of predictive accuracy exhibited by consensus forecasts when using data partitioning approaches may not reflect their performance when predicting temporally independent data. We contend that consensus forecasts may not represent the best approach for predicting species distributions under future climatic change, as they may not provide superior predictive accuracy in novel temporal domains compared to traditional modeling approaches that more readily lend themselves to ecological interpretation of model structure.

Automatic image analysis and spot classification for detection of fruit fly infestation in hyperspectral images of mangoes

Fruit fly infestation of mangos is a major concern for growers and exporters, leading to requirements for quarantine treatments such as vapor heat treatment or irradiation and subsequent reduction in quality and consumer acceptance. An on-line method for detection and removal of infested fruit would thus benefit producers and consumers. An algorithm has been developed to identify spots generated in hyperspectral images of mangoes infested with fruit fly larvae. The algorithm incorporates background removal, application of a Gaussian blur, thresholding, and particle count analysis to identify locations of infestations. Each of the four algorithm steps involves adjustable parameters which were iteratively tested to find the optimal combination for detection in terms of false positive and false negative results. For algorithm parameters selected to minimize false negative results, a false negative error rate of 1.0% was achieved with 11.1% false positive error and 6.0% overall error in heavily infested samples. For the same sample set, the lowest overall error rate achieved was 2.0%, with 1.0% false positive and 3.0% false negative. For samples with lower infestation rates, the error rates were much higher, the lowest overall error being 12.3%. This therefore demonstrates the feasibility of hyperspectral imaging for fruit fly detection while highlighting the need for technology with improved resolution and signal to noise ratio to allow detection of single larvae.

Complete genome phasing of family quartet by combination of genetic, physical and population-based phasing analysis.

Phased genome maps are important to understand genetic and epigenetic regulation and disease mechanisms, particularly parental imprinting defects. Phasing is also critical to assess the functional consequences of genetic variants, and to allow precise definition of haplotype blocks which is useful to understand gene-flow and genotype-phenotype association at the population level. Transmission phasing by analysis of a family quartet allows the phasing of 95% of all variants as the uniformly heterozygous positions cannot be phased. Here, we report a phasing method based on a combination of transmission analysis, physical phasing by pair-end sequencing of libraries of staggered sizes and population-based analysis. Sequencing of a healthy Caucasians quartet at 120x coverage and combination of physical and transmission phasing yielded the phased genotypes of about 99.8% of the SNPs, indels and structural variants present in the quartet, a phasing rate significantly higher than what can be achieved using any single phasing method. A false positive SNP error rate below 10*E-7 per genome and per base was obtained using a combination of filters. We provide a complete list of SNPs, indels and structural variants, an analysis of haplotype block sizes, and an analysis of the false positive and negative variant calling error rates. Improved genome phasing and family sequencing will increase the power of genome-wide sequencing as a clinical diagnosis tool and has myriad basic science applications.

Methods Matter: Rethinking Inventor Disambiguation with Classification & Labeled Inventor Records

Existing record linkage methods often use ad-hoc weights, thresholds, and decision rules that may lead to unwanted and systematic errors in the results. We use a unique set of 98,762 labeled inventor records (i.e. records for which we know the true identification of the individual or entity) obtained from 824 USPTO optoelectronics inventors to test the accuracy of these existing methods. We then develop a new algorithm that is a variant of the Random Forests classification method to predict whether or not pairs of USPTO inventor records match. Our new approach to inventor disambiguation - which we call Conditional Forest of Random Forests - reduces false positive and false negative error rates by 84.5% and 92.7% respectively, over existing USPTO inventor disambiguation algorithms on our set of labeled inventor records. Unlike the existing algorithms, our errors do not occur systematically. The substantial reduction in and less systematic occurrence of disambiguation error suggests that research using the results of existing disambiguation methods should be revisited, as the systematic bias of the errors in these disambiguation results may affect the conclusions of these and subsequent studies.

A Quantitative Analysis of Torn and Cut Duct Tape Physical End Matching

Forensic scientists are often asked to physically compare duct tape samples found in association with criminal activity. This study was designed to statistically evaluate the error and accuracy rates associated with duct tape physical end matching. The experimental design consisted of a blind study in which three researchers independently analyzed eight types of tape subjected to four methods of separation. The lowest mean accuracy observed was 98.15%, the highest mean false-positive rate observed was 3.33%, and the highest mean false-negative rate was 2.67%. Overall, high accuracy with low false-positive and false-negative error rates were observed. This study confirms the use of physical end matching in identifying duct tape samples as matching or nonmatching and that the differences between analysts, brands, tape grades, tape color, and methods of separation have varying contributions to misidentifications and inconclusive results. This study also demonstrates the importance of peer review in duct tape analysis.

A Naturally Occurring Mutation Within the Probe-Binding Region Compromises a Molecular-Based West Nile Virus Surveillance Assay for Mosquito Pools (Diptera: Culicidae)

A naturally occurring mutation was detected within the probe binding region targeting the envelope gene sequence of West Nile virus used in real-time polymerase chain reaction assays to test mosquito pools and other samples. A single C-->T transition 6nt from the 5' end of the 16mer in the envelope gene probe-binding region at genomic position 1,194 reduced assay sensitivity. The mutation first was detected in 2009 and persisted at a low prevalence into 2011. The mutation caused a 0.4% false negative error rate during 2011. These data emphasized the importance of confirmational testing and redundancy in surveillance systems relying on highly specific nucleic acid detection platforms.

Experimental investigation of false positive errors in auditory species occurrence surveys

False positive errors are a significant component of many ecological data sets, which in combination with false negative errors, can lead to severe biases in conclusions about ecological systems. We present results of a field experiment where observers recorded observations for known combinations of electronically broadcast calling anurans under conditions mimicking field surveys to determine species occurrence. Our objectives were to characterize false positive error probabilities for auditory methods based on a large number of observers, to determine if targeted instruction could be used to reduce false positive error rates, and to establish useful predictors of among-observer and among-species differences in error rates. We recruited 31 observers, ranging in abilities from novice to expert, who recorded detections for 12 species during 180 calling trials (66,960 total observations). All observers made multiple false positive errors, and on average 8.1% of recorded detections in the experiment were false positive errors. Additional instruction had only minor effects on error rates. After instruction, false positive error probabilities decreased by 16% for treatment individuals compared to controls with broad confidence interval overlap of 0 (95% CI:--46 to 30%). This coincided with an increase in false negative errors due to the treatment (26%;--3 to 61%). Differences among observers in false positive and in false negative error rates were best predicted by scores from an online test and a self-assessment of observer ability completed prior to the field experiment. In contrast, years of experience conducting call surveys was a weak predictor of error rates. False positive errors were also more common for species that were played more frequently but were not related to the dominant spectral frequency of the call. Our results corroborate other work that demonstrates false positives are a significant component of species occurrence data collected by auditory methods. Instructing observers to only report detections they are completely certain are correct is not sufficient to eliminate errors. As a result, analytical methods that account for false positive errors will be needed, and independent testing of observer ability is a useful predictor for among-observer variation in observation error rates.

Bad statistical practice in pharmacology (and other basic biomedical disciplines): you probably don't knowP

Statistical analysis is universally used in the interpretation of the results of basic biomedical research, being expected by referees and readers alike. Its role in helping researchers to make reliable inference from their work and its contribution to the scientific process cannot be doubted, but can be improved. There is a widespread and pervasive misunderstanding of P-values that limits their utility as a guide to inference, and a change in the manner in which P-values are specified and interpreted will lead to improved outcomes. This paper explains the distinction between Fisher's P-values, which are local indices of evidence against the null hypothesis in the results of a particular experiment, and Neyman-Pearson α levels, which are global rates of false positive errors from unrelated experiments taken as an aggregate. The vast majority of papers published in pharmacological journals specify P-values, either as exact-values or as being less than a value (usually 0.05), but they are interpreted in a hybrid manner that detracts from their Fisherian role as indices of evidence without gaining the control of false positive and false negative error rate offered by a strict Neyman-Pearson approach. An informed choice between those approaches offers substantial advantages to the users of statistical tests over the current accidental hybrid approach.

Methods Matter: Revamping Inventor Disambiguation Algorithms with Classification Models and Labeled Inventor Records

To date, methods used to disambiguate inventors in the United States Patent and Trademark Office (USPTO) database have been rule- and threshold-based (requiring and leveraging expert knowledge) or semi-supervised algorithms trained on statistically-generated artificial labels. Using a large, hand-disambiguated set of 98,762 labeled USPTO inventor records from the field of optoelectronics consisting of four sub-samples of inventors with varying characteristics (Akinsanmi et al. 2014) and a second large, hand-disambiguated set of 53,378 labeled inventor records corresponding to a subset of academics in the life sciences (Azoulay et al. 2012), we provide the first supervised learning approach for USPTO inventor disambiguation. Using these two sets of inventor records, we also provide extensive evaluations of both our algorithm and three examples of prior approaches to USPTO disambiguation arguably representative of the range of approaches used to-date. We show that the three past disambiguation algorithms we evaluate demonstrate biases depending on the feature distribution of the target disambiguation population. Both the rule- and threshold-based methods and the semi-supervised approach perform poorly (10-22% false negative error rates) on a random sample of optoelectronics inventors – arguably the closest of our sub-samples to what might be expected of the majority of inventors in the USPTO (based on disambiguation-relevant metrics). The supervised learning approach, using random forests and trained on our labeled optoelectronics dataset, consistently maintains error rates below 3% across all of our available samples. We make public both our labeled optoelectronics inventor records and our code to build supervised learning models and disambiguate inventors. Our code also allows users to implement supervised learning approaches with their own representative labeled training data.

An Early Fire Detection Algorithm Using IP Cameras

The presence of smoke is the first symptom of fire; therefore to achieve early fire detection, accurate and quick estimation of the presence of smoke is very important. In this paper we propose an algorithm to detect the presence of smoke using video sequences captured by Internet Protocol (IP) cameras, in which important features of smoke, such as color, motion and growth properties are employed. For an efficient smoke detection in the IP camera platform, a detection algorithm must operate directly in the Discrete Cosine Transform (DCT) domain to reduce computational cost, avoiding a complete decoding process required for algorithms that operate in spatial domain. In the proposed algorithm the DCT Inter-transformation technique is used to increase the detection accuracy without inverse DCT operation. In the proposed scheme, firstly the candidate smoke regions are estimated using motion and color smoke properties; next using morphological operations the noise is reduced. Finally the growth properties of the candidate smoke regions are furthermore analyzed through time using the connected component labeling technique. Evaluation results show that a feasible smoke detection method with false negative and false positive error rates approximately equal to 4% and 2%, respectively, is obtained.

Detecting Spam Zombies by Monitoring Outgoing Messages

Compromised machines are one of the key security threats on the Internet; they are often used to launch various security attacks such as spamming and spreading malware, DDoS, and identity theft. Given that spamming provides a key economic incentive for attackers to recruit the large number of compromised machines, we focus on the detection of the compromised machines in a network that are involved in the spamming activities, commonly known as spam zombies. We develop an effective spam zombie detection system named SPOT by monitoring outgoing messages of a network. SPOT is designed based on a powerful statistical tool called Sequential Probability Ratio Test, which has bounded false positive and false negative error rates. In addition, we also evaluate the performance of the developed SPOT system using a two-month e-mail trace collected in a large US campus network. Our evaluation studies show that SPOT is an effective and efficient system in automatically detecting compromised machines in a network. For example, among the 440 internal IP addresses observed in the e-mail trace, SPOT identifies 132 of them as being associated with compromised machines. Out of the 132 IP addresses identified by SPOT, 126 can be either independently confirmed (110) or highly likely (16) to be compromised. Moreover, only seven internal IP addresses associated with compromised machines in the trace are missed by SPOT. In addition, we also compare the performance of SPOT with two other spam zombie detection algorithms based on the number and percentage of spam messages originated or forwarded by internal machines, respectively, and show that SPOT outperforms these two detection algorithms.

A completely in vitro ultrahigh-throughput droplet-based microfluidic screening system for protein engineering and directed evolution

In vitro screening systems based on the coupled transcription and translation of genes using cell-free systems have a number of attractive features for protein engineering and directed evolution. We present a completely in vitro ultrahigh-throughput screening platform using droplet-based microfluidics. Single genes are compartmentalized in aqueous droplets, dispersed in inert carrier oil, and amplified using the polymerase chain reaction (PCR). After amplification, the droplets, now containing 30,000 copies of each gene, are fused one-to-one with droplets containing a cell-free coupled transcription-translation (IVTT) system and the reagents for a fluorogenic assay. Fluorescence-activated electrocoalescence with an aqueous stream is then used to selectively recover genes from droplets containing the desired activity. We demonstrate, by selecting mixtures of lacZ genes encoding the enzyme β-galactosidase and lacZmut genes encoding an inactive variant, that this system can sort at 2000 droplets s(-1): lacZ genes were enriched 502-fold from a 1 : 100 molar ratio of lacZ : lacZmut genes. Indeed, the false positive and false negative error rates were both <0.004 and the results indicate that enrichment is not limited by the sorting efficiency, but by the co-encapsulation of multiple genes in droplets, which is described by the Poisson distribution. Compared to screening using microtiter plate-based systems, the volume and cost of PCR and IVTT reagents are reduced by almost 10(5)-fold, allowing the screening of 10(6) genes using only 150 μL of reagents.

Adaptive thresholding for reliable topological inference in single subject fMRI analysis

Single subject fMRI has proved to be a useful tool for mapping functional areas in clinical procedures such as tumor resection. Using fMRI data, clinicians assess the risk, plan and execute such procedures based on thresholded statistical maps. However, because current thresholding methods were developed mainly in the context of cognitive neuroscience group studies, most single subject fMRI maps are thresholded manually to satisfy specific criteria related to single subject analyzes. Here, we propose a new adaptive thresholding method which combines Gamma-Gaussian mixture modeling with topological thresholding to improve cluster delineation. In a series of simulations we show that by adapting to the signal and noise properties, the new method performs well in terms of total number of errors but also in terms of the trade-off between false negative and positive cluster error rates. Similarly, simulations show that adaptive thresholding performs better than fixed thresholding in terms of over and underestimation of the true activation border (i.e., higher spatial accuracy). Finally, through simulations and a motor test–retest study on 10 volunteer subjects, we show that adaptive thresholding improves reliability, mainly by accounting for the global signal variance. This in turn increases the likelihood that the true activation pattern can be determined offering an automatic yet flexible way to threshold single subject fMRI maps.

Error rates in reporting prostatic core biopsies

To evaluate the false-negative and false-positive error rates both in a screening and a non-screening population. A total of 4192 prostatic biopsies were reported in a 6-year period by 15 consultant histopathologists, two of whom had an interest in uropathology and were deemed to be specialists (J.O. and C.S.). All biopsies were reviewed prior to the multidisciplinary team (MDT) meeting. The overall false-negative rate was 1.7% (screening 2.1%, non-screening 1.5%). The overall false-positive rate was 0.5% (screening 0.9%, non-screening 0.4%). These error rates varied among pathologists, with the false-negative rate ranging from 0% to 9.3%, and the false-positive rate ranging from 0% to 3.8%. The false-negative rate was three times greater than the false-positive rate, showing that detection of significant pathology is far greater in the negative biopsies. More errors occurred in the screening population than in the non-screening population. The consultants making the most errors were non-specialists, but the specialists also made false-negative errors, suggesting that just using specialist reporting alone would not have eradicated errors.

An image differencing method for interface level detection in separation cells

Bitumen extraction from oil sands is carried out in large separation cells using a water-based extraction process. Controlling the Bitumen-froth and Middlings interface in these cells at an optimum level provides significant economical and environmental benefits. Traditional sensors are not reliable in estimating this interface level and novel vision-based sensors have been developed previously to overcome this problem. These sensors estimate the interface level and its confidence for separation cells with a single sight glass. The confidence value only represents the turbidity of the interface and hence cannot be used for control decision making in all process conditions. The current work describes an image differencing algorithm for interface level detection which also facilitates the computation of a confidence estimate that is accurate in most process situations. The confidence value is computed based on noise statistics, an appropriately chosen edge detection method and a change detection algorithm. Another significant advantage of the algorithm is that both the interface level and the confidence estimation procedures can be extended in a straight-forward manner to handle the presence of multiple sight glasses. Off-line results show that the algorithm accurately detects the interface level in normal process conditions (with high confidence values) and outputs correct confidence values in other situations with very low false-positive and false-negative error rates. On-line industrial implementation results show that the vision sensor tracks the interface level very closely and results in significant automation of plant.

Comparison of FST outlier tests for SNP loci under selection

Genome scans with many genetic markers provide the opportunity to investigate local adaptation in natural populations and identify candidate genes under selection. In particular, SNPs are dense throughout the genome of most organisms and are commonly observed in functional genes making them ideal markers to study adaptive molecular variation. This approach has become commonly employed in ecological and population genetics studies to detect outlier loci that are putatively under selection. However, there are several challenges to address with outlier approaches including genotyping errors, underlying population structure and false positives, variation in mutation rate and limited sensitivity (false negatives). In this study, we evaluated multiple outlier tests and their type I (false positive) and type II (false negative) error rates in a series of simulated data sets. Comparisons included simulation procedures (FDIST2, ARLEQUIN v.3.5 and BAYESCAN) as well as more conventional tools such as global F(ST) histograms. Of the three simulation methods, FDIST2 and BAYESCAN typically had the lowest type II error, BAYESCAN had the least type I error and Arlequin had highest type I and II error. High error rates in Arlequin with a hierarchical approach were partially because of confounding scenarios where patterns of adaptive variation were contrary to neutral structure; however, Arlequin consistently had highest type I and type II error in all four simulation scenarios tested in this study. Given the results provided here, it is important that outlier loci are interpreted cautiously and error rates of various methods are taken into consideration in studies of adaptive molecular variation, especially when hierarchical structure is included.

The consequences of using indirect signs that decay to determine species' occupancy

Statistical models of species' distributions rely on data on species' occupancy, or use, of sites across space and/or time. For rare or cryptic species, indirect signs, such as dung, may be the only realistic means of determining their occupancy status across broad spatial extents. However, the consequences of sign decay for errors in estimates of occupancy have not previously been considered. If signs decay very rapidly, then false-negative errors may occur because signs at an occupied site have decayed by the time it is surveyed. On the other hand, if signs decay very slowly, false-positive errors may occur because signs remain present at sites that are no longer occupied. We addressed this issue by quantifying, as functions of sign decay and accumulation rates: 1) the false-negative error rate due to sign decay and, 2) the expected time interval prior to a survey within which signs indicate the species was present; as this time interval increases, false-positives become more likely. We then applied this to the specific example of koala Phascolarctos cinereus occupancy derived from faecal pellet surveys using data on faecal pellet decay rates. We show that there is a clear trade-off between false-negative error rates and the potential for false-positive errors. For the koala case study, false-negative errors were low on average and the expected time interval prior to surveys that detected pellets indicate the species was present within less than 2-3 yr. However, these quantities showed quite substantial spatial variation that could lead to biased parameter estimates for distribution models based on faecal pellet surveys. This highlights the importance of observation errors arising from sign decay and we suggest some modifications to existing methods to deal with this issue.

Establishing a match criterion in forensic comparison analysis of float glass using laser ablation inductively coupled plasma mass spectrometry

In the field of forensic analysis of glass the aim in most cases is to prove or disprove the common origin of two (or more) sources of glass. Glass isolated from clothing of a suspect is compared to broken glass originated from a scene of crime in order to link these together. Based on the given merits, LA-ICP-MS is an excellent tool for the analysis of glass fragments. In this paper, a compromise match criterion for glass casework is established and optimised in order to produce low Type 1 (false negative) error rates and low Type 2 (false positive) error rates at the same time. Glass from two sets (fragments originating from the same float glass pane and fragments originating from different float glass of global origin) were analysed using laser ablation-inductively coupled plasma-mass spectrometry. Comparisons of the quantitative results for 18 element concentrations (Li, Na, Mg, Al, K, Ca, Ti, Mn, Fe, Rb, Sr, Zr, Ba, La, Ce, Nd, Hf, and Pb) were performed using different statistical methods. The best results regarding a compromise match criterion for glass casework were achieved using a modified 4 sigma match criterion based on fixed relative standard deviations, or using t-tests modified analogously by applying the same fixed relative standard deviations. The modified 4 sigma criterion was chosen to be applied in routine casework and has been successfully applied in more than 70 court cases.