Estimates Of False Positive Rates Research Articles

Considering the effect of correlation between the channels in multivariate detectors for evoked responses in the electroencephalogram

Evoked potentials refer to the small responses in the electroencephalogram (EEG), resulting from repeated or continuous auditory, visual or other stimuli. The detection of these responses is most commonly based on visual inspection by experienced professionals. To overcome the inherent subjectivity, automated (usually statistical) methods can be used, which aim to find the statistical significance of some parameter(s) calculated from the recorded signals, based on the null-hypothesis of 'no stimulus response is present'. These tests need to rely on assumptions about the statistical distributions of the signals and their parameters which in many cases allows p-values to be calculated. For some convenient parameters, however, closed form mathematical solutions are intractable and simulations are often required to find p-values (or critical values). The use of multichannel recordings can increase the sensitivity of tests compared to single-channel EEG, permitting the detection of responses in shorter recordings. This can be achieved by simple extensions of originally single channel methods, such as magnitude-squared coherence, component synchrony measure or the spectral F-test. However, the correlation between EEG channels can lead to false-positive rates that are difficult to predict or control. In this paper we present a modified parametric bootstrap method that permits the accurate estimation of false positive rates (p-values and critical values) for multichannel extensions of univariate objective evoked response techniques, with concomitant increases in sensitivity over conventional single channel methods. Illustrative results with evoked potentials from repeated photic stimulation are provided, with multivariate magnitude-squared coherence demonstrating the most robust performance.

Integrative analysis of liver-specific non-coding regulatory SNPs associated with the risk of coronary artery disease

Genetic factors underlying coronary artery disease (CAD) have been widely studied using genome-wide association studies (GWASs). However, the functional understanding of the CAD loci has been limited by the fact that a majority of GWAS variants are located within non-coding regions with no functional role. High cholesterol and dysregulation of the liver metabolism such as non-alcoholic fatty liver disease confer an increased risk of CAD. Here, we studied the function of non-coding single-nucleotide polymorphisms in CAD GWAS loci located within liver-specific enhancer elements by identifying their potential target genes using liver cis-eQTL analysis and promoter Capture Hi-C in HepG2 cells. Altogether, 734 target genes were identified of which 121 exhibited correlations to liver-related traits. To identify potentially causal regulatory SNPs, the allele-specific enhancer activity was analyzed by (1) sequence-based computational predictions, (2) quantification of allele-specific transcription factor binding, and (3) STARR-seq massively parallel reporter assay. Altogether, our analysis identified 1,277 unique SNPs that display allele-specific regulatory activity. Among these, susceptibility enhancers near important cholesterol homeostasis genes (APOB, APOC1, APOE, and LIPA) were identified, suggesting that altered gene regulatory activity could represent another way by which genetic variation regulates serum lipoprotein levels. Using CRISPR-based perturbation, we demonstrate how the deletion/activation of a single enhancer leads to changes in the expression of many target genes located in a shared chromatin interaction domain. Our integrative genomics approach represents a comprehensive effort in identifying putative causal regulatory regions and target genes that could predispose to clinical manifestation of CAD by affecting liver function.

DetEdit: A graphical user interface for annotating and editing events detected in long-term acoustic monitoring data.

Passive acoustic monitoring has become an important data collection method, yielding massive datasets replete with biological, environmental and anthropogenic information. Automated signal detectors and classifiers are needed to identify events within these datasets, such as the presence of species-specific sounds or anthropogenic noise. These automated methods, however, are rarely a complete substitute for expert analyst review. The ability to visualize and annotate acoustic events efficiently can enhance scientific insights from large, previously intractable datasets. A MATLAB-based graphical user interface, called DetEdit, was developed to accelerate the editing and annotating of automated detections from extensive acoustic datasets. This tool is highly-configurable and multipurpose, with uses ranging from annotation and classification of individual signals or signal-clusters and evaluation of signal properties, to identification of false detections and false positive rate estimation. DetEdit allows users to step through acoustic events, displaying a range of signal features, including time series of received levels, long-term spectral averages, time intervals between detections, and scatter plots of peak frequency, RMS, and peak-to-peak received levels. Additionally, it displays either individual, or averaged sound pressure waveforms, and power spectra within each acoustic event. These views simultaneously provide analysts with signal-level detail and encounter-level context. DetEdit creates datasets of signal labels for further analyses, such as training classifiers and quantifying occurrence, abundances, or trends. Although designed for evaluating underwater-recorded odontocete echolocation click detections, DetEdit can be adapted to almost any stereotyped impulsive signal. Our software package complements available tools for the bioacoustic community and is provided open source at https://github.com/MarineBioAcousticsRC/DetEdit.

Performance on bedside tests of attention and organized thinking in patients with dementia free from delirium.

This cross-sectional study was conducted in outpatients and institutionalized patients without delirium representing a wide spectrum of severity of cognitive impairments. Patients with dementia or a cognitive disorder according to DSM IV criteria, after exclusion of (suspected) delirium according to DSM IV criteria. Tests for inattention and disorganized thinking from the CAM-ICU were assessed. The sample included 163 patients (mean age 83 years (SD 6; 64% women)), with Alzheimer's disease as most prevalent (45%) diagnosis and a mean MMSE-score of 16.8 (SD 7.5). False positive rates of the test of attention varied from 0.04 in patients with normal to borderline cognitive function to 0.8 in those with severe dementia. The false positive rate of the test of disorganized thinking was zero in the normal to borderline group, increasing to 0.67 in patients with severe dementia. When combining test results false positive rates decreased to 0.03 in patients with MMSE scores above 9. Use of simple bedside tests of attention and organized thinking for the clinical diagnosis of DSD will result in high rates of false positive observations if used regardless of the severity of dementia. However, if test results are combined they may be useful to exclude DSD in patients with minimal to moderate degrees of dementia, but not in the severe group.

Ranking metrics in gene set enrichment analysis: do they matter?

BackgroundThere exist many methods for describing the complex relation between changes of gene expression in molecular pathways or gene ontologies under different experimental conditions. Among them, Gene Set Enrichment Analysis seems to be one of the most commonly used (over 10,000 citations). An important parameter, which could affect the final result, is the choice of a metric for the ranking of genes. Applying a default ranking metric may lead to poor results.Methods and resultsIn this work 28 benchmark data sets were used to evaluate the sensitivity and false positive rate of gene set analysis for 16 different ranking metrics including new proposals. Furthermore, the robustness of the chosen methods to sample size was tested. Using k-means clustering algorithm a group of four metrics with the highest performance in terms of overall sensitivity, overall false positive rate and computational load was established i.e. absolute value of Moderated Welch Test statistic, Minimum Significant Difference, absolute value of Signal-To-Noise ratio and Baumgartner-Weiss-Schindler test statistic. In case of false positive rate estimation, all selected ranking metrics were robust with respect to sample size. In case of sensitivity, the absolute value of Moderated Welch Test statistic and absolute value of Signal-To-Noise ratio gave stable results, while Baumgartner-Weiss-Schindler and Minimum Significant Difference showed better results for larger sample size. Finally, the Gene Set Enrichment Analysis method with all tested ranking metrics was parallelised and implemented in MATLAB, and is available at https://github.com/ZAEDPolSl/MrGSEA.ConclusionsChoosing a ranking metric in Gene Set Enrichment Analysis has critical impact on results of pathway enrichment analysis. The absolute value of Moderated Welch Test has the best overall sensitivity and Minimum Significant Difference has the best overall specificity of gene set analysis. When the number of non-normally distributed genes is high, using Baumgartner-Weiss-Schindler test statistic gives better outcomes. Also, it finds more enriched pathways than other tested metrics, which may induce new biological discoveries.

A novel approach to assess the treatment response using Gaussian random field in PET.

The assessment of early therapeutic response to anticancer therapy is vital for treatment planning and patient management in clinic. With the development of personal treatment plan, the early treatment response, especially before any anatomically apparent changes after treatment, becomes urgent need in clinic. Positron emission tomography (PET) imaging serves an important role in clinical oncology for tumor detection, staging, and therapy response assessment. Many studies on therapy response involve interpretation of differences between two PET images, usually in terms of standardized uptake values (SUVs). However, the quantitative accuracy of this measurement is limited. This work proposes a statistically robust approach for therapy response assessment based on Gaussian random field (GRF) to provide a statistically more meaningful scale to evaluate therapy effects. The authors propose a new criterion for therapeutic assessment by incorporating image noise into traditional SUV method. An analytical method based on the approximate expressions of the Fisher information matrix was applied to model the variance of individual pixels in reconstructed images. A zero mean unit variance GRF under the null hypothesis (no response to therapy) was obtained by normalizing each pixel of the post-therapy image with the mean and standard deviation of the pretherapy image. The performance of the proposed method was evaluated by Monte Carlo simulation, where XCAT phantoms (128(2) pixels) with lesions of various diameters (2-6 mm), multiple tumor-to-background contrasts (3-10), and different changes in intensity (6.25%-30%) were used. The receiver operating characteristic curves and the corresponding areas under the curve were computed for both the proposed method and the traditional methods whose figure of merit is the percentage change of SUVs. The formula for the false positive rate (FPR) estimation was developed for the proposed therapy response assessment utilizing local average method based on random field. The accuracy of the estimation was validated in terms of Euler distance and correlation coefficient. It is shown that the performance of therapy response assessment is significantly improved by the introduction of variance with a higher area under the curve (97.3%) than SUVmean (91.4%) and SUVmax (82.0%). In addition, the FPR estimation serves as a good prediction for the specificity of the proposed method, consistent with simulation outcome with ∼1 correlation coefficient. In this work, the authors developed a method to evaluate therapy response from PET images, which were modeled as Gaussian random field. The digital phantom simulations demonstrated that the proposed method achieved a large reduction in statistical variability through incorporating knowledge of the variance of the original Gaussian random field. The proposed method has the potential to enable prediction of early treatment response and shows promise for application to clinical practice. In future work, the authors will report on the robustness of the estimation theory for application to clinical practice of therapy response evaluation, which pertains to binary discrimination tasks at a fixed location in the image such as detection of small and weak lesion.

Use of the ecf1 Gene To Detect Shiga Toxin–Producing Escherichia coli in Beef Samples

Escherichia coli O157:H7 and six serovars (O26, O103, O121, O111, O145, and O45) are frequently implicated in severe clinical illness worldwide. Standard testing methods using stx, eae, and O serogroup–specific gene sequences for detecting the top six non-O157 STEC bear the disadvantage that these genes may reside, independently, in different nonpathogenic organisms, leading to false-positive results. The ecf operon has previously been identified in the large enterohemolysin-encoding plasmid of eae-positive Shiga toxin–producing E. coli (STEC). Here, we explored the utility of the ecf operon as a single marker to detect eae-positive STEC from pure broth and primary meat enrichments. Analysis of 501 E. coli isolates demonstrated a strong correlation (99.6%) between the presence of the ecf1 gene and the combined presence of stx, eae, and ehxA genes. Two large studies were carried out to determine the utility of an ecf1 detection assay to detect non-O157 STEC strains in enriched meat samples in comparison to the results using the U.S. Department of Agriculture Food Safety and Inspection Service (FSIS) method that detects stx and eae genes. In ground beef samples (n = 1,065), the top six non-O157 STEC were detected in 4.0% of samples by an ecf1 detection assay and in 5.0% of samples by the stx- and eae-based method. In contrast, in beef samples composed largely of trim (n = 1,097), the top six non-O157 STEC were detected at 1.1% by both methods. Estimation of false-positive rates among the top six non-O157 STEC revealed a lower rate using the ecf1 detection method (0.5%) than using the eae and stx screening method (1.1%). Additionally, the ecf1 detection assay detected STEC strains associated with severe illness that are not included in the FSIS regulatory definition of adulterant STEC.

Variability in definitions of transaminase upper limit of the normal impacts the APRI performance as a biomarker of fibrosis in patients with chronic hepatitis C: “APRI c’est fini ?”

The aspartate aminotransferase platelet ratio index (APRI) is a validated, non-patented blood test for diagnosing fibrosis or cirrhosis in patients with chronic hepatitis C. We assess the impact of two limitations, the variability of the upper limit of normal for aspartate aminotransferase (AST-ULN) and the risk of overestimating fibrosis stage due to necroinflammatory activity. The variability of AST-ULN was assessed by an overview of the literature and an assessment of AST-ULN in 2 control populations 7521 healthy volunteers and 393 blood donors. We assessed the impact of AST-ULN variability on APRI performance for estimating fibrosis prevalence and on the Obuchowski measure using individual data of 1651 patients with APRI, FibroTest and biopsy. The overview, and the analysis of the control populations found that ULN-AST ranged from 26 to 49 IU/L according to gender, body mass index and serum cholesterol. When this AST-ULN variability was applied to the chronic hepatitis group, the prevalence of advanced fibrosis and cirrhosis as presumed by APRI varied (P<0.001) from 34.7% to 68.5%, and from 11.4% to 32.3%, respectively. This spectrum effect induced variability in APRI performance, which could be similar 0.862 (if AST-ULN=26 IU/L) or lower 0.820 (AST-ULN≥30IU/L) than the stable FibroTest performance (0.867; P=0.35 and P<0.0001 respectively). When applied to 18 acute hepatitis C patients, the rate of false positives of APRI varied from 0% to 61% due to AST-ULN. The AST-ULN variability is high highly associated with the variability of metabolic risk factors between the different control groups. This variability induces a spectrum effect, which could cause misleading interpretations of APRI performance for the staging of fibrosis, comparisons of APRI with other non-invasive tests, and estimates of false positive rate.

Analysis of Protein–Protein Interactions and Proteomic Profiles of Normal Human Lenses

Purpose: To investigate proteomic profiles of normal human lenses and their key proteins in protein–protein interactions (PPIs).Materials and Methods: Water-soluble and water-insoluble proteins extracted from human lenses were first separated by one-dimensional sodium dodecyl sulfate polyacrylamide gel, and then in-gel digested with trypsin into peptides eluted by reversed-phase high-performance liquid chromatography. The eluted peptides were analyzed by linear ion trap tandem mass spectrometry (MS/MS). The raw data was filtered by TurboSEQUEST algorithm. The reverse database was used for peptide false-positive rate estimation. A network chart was constructed by the identified lens PPIs in accordance with interaction database systems.Results: From normal human lenses 339 proteins in total were identified, including many formerly unidentified low-abundance proteins. Key proteins we recognized included plectin, actin, spectrin (α, β), vimentin, 14-3-3 protein (β/α, ζ/δ, ϵ, γ, η), TSC2, guanine nucleotide-releasing protein, laminin γ, mitogen-activated protein kinase, α-A-crystallin, heat-shock protein (α, β), glyceraldehyde 3-phosphate dehydrogenase, and collagen IV α.Conclusions: Key proteins of normal human lenses were studied by constructing a network chart of the identified lens PPIs. The results suggest that linear ion trap MS/MS is an effective tool for detecting low-abundance proteins of human lenses. This study provides valuable data for further proteomic research of the human lens development and lens diseases.

Dinucleotide controlled null models for comparative RNA gene prediction

BackgroundComparative prediction of RNA structures can be used to identify functional noncoding RNAs in genomic screens. It was shown recently by Babak et al. [BMC Bioinformatics. 8:33] that RNA gene prediction programs can be biased by the genomic dinucleotide content, in particular those programs using a thermodynamic folding model including stacking energies. As a consequence, there is need for dinucleotide-preserving control strategies to assess the significance of such predictions. While there have been randomization algorithms for single sequences for many years, the problem has remained challenging for multiple alignments and there is currently no algorithm available.ResultsWe present a program called SISSIz that simulates multiple alignments of a given average dinucleotide content. Meeting additional requirements of an accurate null model, the randomized alignments are on average of the same sequence diversity and preserve local conservation and gap patterns. We make use of a phylogenetic substitution model that includes overlapping dependencies and site-specific rates. Using fast heuristics and a distance based approach, a tree is estimated under this model which is used to guide the simulations. The new algorithm is tested on vertebrate genomic alignments and the effect on RNA structure predictions is studied. In addition, we directly combined the new null model with the RNAalifold consensus folding algorithm giving a new variant of a thermodynamic structure based RNA gene finding program that is not biased by the dinucleotide content.ConclusionSISSIz implements an efficient algorithm to randomize multiple alignments preserving dinucleotide content. It can be used to get more accurate estimates of false positive rates of existing programs, to produce negative controls for the training of machine learning based programs, or as standalone RNA gene finding program. Other applications in comparative genomics that require randomization of multiple alignments can be considered.AvailabilitySISSIz is available as open source C code that can be compiled for every major platform and downloaded here: .

A nonparametric model for quality control of database search results in shotgun proteomics

BackgroundAnalysis of complex samples with tandem mass spectrometry (MS/MS) has become routine in proteomic research. However, validation of database search results creates a bottleneck in MS/MS data processing. Recently, methods based on a randomized database have become popular for quality control of database search results. However, a consequent problem is the ignorance of how to combine different database search scores to improve the sensitivity of randomized database methods.ResultsIn this paper, a multivariate nonlinear discriminate function (DF) based on the multivariate nonparametric density estimation technique was used to filter out false-positive database search results with a predictable false positive rate (FPR). Application of this method to control datasets of different instruments (LCQ, LTQ, and LTQ/FT) yielded an estimated FPR close to the actual FPR. As expected, the method was more sensitive when more features were used. Furthermore, the new method was shown to be more sensitive than two commonly used methods on 3 complex sample datasets and 3 control datasets.ConclusionUsing the nonparametric model, a more flexible DF can be obtained, resulting in improved sensitivity and good FPR estimation. This nonparametric statistical technique is a powerful tool for tackling the complexity and diversity of datasets in shotgun proteomics.

A new strategy to filter out false positive identifications of peptides in SEQUEST database search results

Based on the randomized database method and a linear discriminant function (LDF) model, a new strategy to filter out false positive matches in SEQUEST database search results is proposed. Given an experiment MS/MS dataset and a protein sequence database, a randomized database is constructed and merged with the original database. Then, all MS/MS spectra are searched against the combined database. For each expected false positive rate (FPR), LDFs are constructed for different charge states and used to filter out the false positive matches from the normal database. In order to investigate the error of FPR estimation, the new strategy was applied to a reference dataset. As a result, the estimated FPR was very close to the actual FPR. While applied to a human K562 cell line dataset, which is a complicated dataset from real sample, more matches could be confirmed than the traditional cutoff-based methods at the same estimated FPR. Also, though most of the results confirmed by the LDF model were consistent with those of PeptideProphet, the LDF model could still provide complementary information. These results indicate that the new method can reliably control the FPR of peptide identifications and is more sensitive than traditional cutoff-based methods.

Mixed group validation: a method to address the limitations of criterion group validation in research on malingering detection.

Mixed group validation (MGV) is offered as an alternative to criterion group validation (CGV) to estimate the true positive and false positive rates of tests and other diagnostic signs. CGV requires perfect confidence about each research participant's status with respect to the presence or absence of pathology. MGV determines diagnostic efficiencies based on group data; knowing an individual's status with respect to pathology is not required. MGV can use relatively weak indicators to validate better diagnostic signs, whereas CGV requires perfect diagnostic signs to avoid error in computing true positive and false positive rates. The process of MGV is explained, and a computer simulation demonstrates the soundness of the procedure. MGV of the Rey 15-Item Memory Test (Rey, 1958) for 723 pre-trial criminal defendants resulted in higher estimates of true positive rates and lower estimates of false positive rates as compared with prior research conducted with CGV. The author demonstrates how MGV addresses all the criticisms Rogers (1997b) outlined for differential prevalence designs in malingering detection research.