Number Of False Positive Signals Research Articles

Performance and Attitudes Toward Real-time Computer-aided Polyp Detection during Colonoscopy in a Large Tertiary Referral Center in the United States.

Computer aided detection (CADe) has been shown to improve polyp detection in clinical trials. Limited data exists on the impact, utilization, and attitudes toward artificial intelligence (AI)-assisted colonoscopy in daily clinical practice. We aimed to evaluate the effectiveness of the first FDA-approved CADe device in the United States and the attitudes toward its implementation. Retrospective analysis of a prospectively maintained database of patients undergoing colonoscopy at a tertiary center in the United States before and after a real-time CADe system was made available. The decision to activate the CADe system was at the discretion of the endoscopist. An anonymous survey was circulated among endoscopy physicians and staff at the beginning and at the conclusion of the study period regarding their attitudes toward AI-assisted colonoscopy. CADe was activated in 52.1% of cases. Compared with historical controls, there was no statistically significant difference in adenomas detected per colonoscopy (APC) (1.08 vs 1.04, p=0.65), even after excluding diagnostic/therapeutic indications and cases where CADe was not activated (1.27 vs 1.17, p=0.45). In addition, there was no statistically significant difference in ADR, median procedure and withdrawal times. Survey results demonstrated mixed attitudes toward AI-assisted colonoscopy, of which main concerns included high number of false positive signals (82.4%), high level of distraction (58.8%), and impression it prolonged procedure time (47.1%). CADe did not improve adenoma detection in daily practice among endoscopists with high baseline ADR. Despite its availability, AI-assisted colonoscopy was only activated in half of cases, and multiple concerns were raised by staff and endoscopists. Future studies will help elucidate the patients and endoscopists that would benefit most from AI-assisted colonoscopy.

Simulating ComBat: how batch correction can lead to the systematic introduction of false positive results in DNA methylation microarray studies

BackgroundSystematic technical effects—also called batch effects—are a considerable challenge when analyzing DNA methylation (DNAm) microarray data, because they can lead to false results when confounded with the variable of interest. Methods to correct these batch effects are error-prone, as previous findings have shown.ResultsHere, we demonstrate how using the R function ComBat to correct simulated Infinium HumanMethylation450 BeadChip (450 K) and Infinium MethylationEPIC BeadChip Kit (EPIC) DNAm data can lead to a large number of false positive results under certain conditions. We further provide a detailed assessment of the consequences for the highly relevant problem of p-value inflation with subsequent false positive findings after application of the frequently used ComBat method. Using ComBat to correct for batch effects in randomly generated samples produced alarming numbers of false discovery rate (FDR) and Bonferroni-corrected (BF) false positive results in unbalanced as well as in balanced sample distributions in terms of the relation between the outcome of interest variable and the technical position of the sample during the probe measurement. Both sample size and number of batch factors (e.g. number of chips) were systematically simulated to assess the probability of false positive findings. The effect of sample size was simulated using n = 48 up to n = 768 randomly generated samples. Increasing the number of corrected factors led to an exponential increase in the number of false positive signals. Increasing the number of samples reduced, but did not completely prevent, this effect.ConclusionsUsing the approach described, we demonstrate, that using ComBat for batch correction in DNAm data can lead to false positive results under certain conditions and sample distributions. Our results are thus contrary to previous publications, considering a balanced sample distribution as unproblematic when using ComBat. We do not claim completeness in terms of reporting all technical conditions and possible solutions of the occurring problems as we approach the problem from a clinician’s perspective and not from that of a computer scientist. With our approach of simulating data, we provide readers with a simple method to assess the probability of false positive findings in DNAm microarray data analysis pipelines.

Evaluation of Early Aberration Reporting System for Dengue Outbreak Detection in Thailand

Thailand is one of the highest-burden countries for dengue infections in the South-East Asia Region of the World Health Organization. The 5-year median is normally used for outbreak detection; however, studies assessing the performance of this indicator against other detection methods are lacking. We, therefore, conducted a descriptive ecological study from a dataset comprised of patient visits to public hospitals for dengue treatment that were reported to the Ministry of Public Health. The aim was to evaluate the performance of an early aberration reporting system (EARS) in detecting dengue outbreaks, compared to using the 5-year median method. During 2003-2015, there were 1,014,201 patient visits and seven reported dengue outbreaks, with the largest occurring in 2013, and six seasonal peaks. The EARS was able to detect all seven dengue outbreaks and six seasonal peaks, including one outbreak that occurred in 2014 which was undetected by the 5-year median. However, EARS cannot provide information on trends, outbreak severity and issues noise signals. Our recommendation was to combine the EARS with the 5-year median method to reduce the number of false positive signals, or use the 5-year median method as a confirmatory tool.

Tree-based scan statistic – Application in manufacturing-related safety signal detection

Background and objectivesOver the last decades, medicinal regulations have been put into place and have considerably improved manufacturing practices. Nevertheless, safety issues may still arise. Using the simulation described in this manuscript, our aim is to develop adequate detection methods for manufacturing-related safety signals, especially in the context of biological products. MethodsPharmaceutical companies record the entire batch genealogies, from seed batches over intermediates to final product (FP) batches. We constructed a hierarchical tree based on this genealogy information and linked it to the spontaneous safety data available for the FP batch numbers. The tree-based scan statistic (TBSS) was used on simulated data as a proof of concept to locate the source that may have subsequently generated an excess of specific adverse events (AEs) within the manufacturing steps, and to evaluate the method’s adjustment for multiple testing. All calculations were performed with a customized program in SAS v9.2. ResultsThe TBSS generated a close to expected number of false positive signals, demonstrating that it adjusted for multiple testing. Overall, the method detected 71% of the simulated signals at the correct production step when a 6-fold increase in reports with AEs of interest (AEOI) was applied, and 31% when a 2-fold increase was applied. The relatively low detection performance may be attributed to the higher granularity associated with the lower levels of the hierarchy, leading to a lack of power and the stringent definition criteria that were applied for a true positive result. ConclusionAs a data-mining method for manufacturing-related safety signal detection, the TBSS may provide advantages over other disproportionality analyses (using batch information) but may benefit from complementary methods (not relaying on batch information). While the method warrants further refinement, it may improve safety signal detection and contribute to improvements in the quality of manufacturing processes.

Translating Research into Improved Patient Care in Pulmonary Arterial Hypertension.

Abstract Despite important advances in its therapeutic management, pulmonary arterial hypertension (PAH) remains an incurable disease. Although numerous drugs exhibited beneficial effects in preclinical settings, only few have reached clinical trial phases, highlighting the challenges of translating preclinical investigations into clinical trials. Potential reasons for delayed PAH drug development may include the inherent limitations of the currently available animal and in vitro models, potential lack of appropriate standardization of the experimental design, regulatory agencies requirements, competing clinical trials and insufficient funding. Although this is not unique to PAH, there is urgency for reducing the number of false positive signals in preclinical studies and optimizing the development of innovative therapeutic targets through performance of clinical trials based on more robust experimental data. The current review discusses the challenges and opportunities in preclinical research to foster drug development in PAH.

Proof of Principle for Mutations Monitoring Using Picoliter-Droplet Digital PCR on DNA and Living Cells: Application to Myelodysplastic Syndromes

IntroductionMyelodysplastic syndromes (MDS) are phenotypically and genotypically heterogeneous diseases with several driver mutations, which are closely related to patient prognosis. Dynamic evolution of mutations reflects the selection of subclones during disease evolution until transformation in secondary acute myeloid leukemia. Thanks to increasing knowledge in gene functions, target drugs are now available in therapeutic. However, questions remain on the impact of such treatments on malignant cells. We have previously investigated the effects of lenalidomide on clonal evolution, by monitoring variant allele frequencies (VAF) using next generation sequencing (NGS) in non-del5q MDS patients (Chesnais et al, Blood 2015). Here, we present a rapid and ultra-sensitive method using picoliter-droplet digital PCR for mutation detection in MDS with ring sideroblasts (RS).Materials and MethodsBone marrow aspirates were obtained from MDS patients included at diagnosis in a multicentric observational trial (PHRC MDS-04, NCT02619565). Three cell lines (HL60, OCI-AML3, UKE-1) were also used to establish the specificity and the sensitivity of assays. Both frozen living cells and extracted DNA were used. Selected samples were screened for mutations in 39 genes by an NGS approach using a Personal Genome Machine® (PGM, ThermoFisher Scientific, Waltham, MA, USA). Primers and probes were designed for Taqman assays based on allelic discrimination of recurrent mutations found in DNMT3A, SF3B1, JAK2 and NRAS genes. For the detection of SF3B1 p.K700E mutation, 3 locked nucleic acids were notably added to the probes to improve specificity. Picoliter-droplet digital PCR was performed on RainDrop® Digital PCR System (RainDance™ Technologies).ResultsAllelic discrimination assays were validated on genomic DNA extracted from cell lines and patient samples harboring or not targeted mutations using the RainDance system. About 5.106 droplets were generated using RainDrop Source. Wild-type (WT) DNA was tested in order to assess false positive signals for each design, characterized by λFP (mean number of false positive signals), limit of blanck (LOB) and limit of detection (LOD) for all experiments. The limit of blanck (LOB) defined here the highest number of droplets corresponding to apparent droplets containing mutated amplicons while testing wild type DNA. The limit of detection (LOD) was the lower number of droplets which can be distinguish from LOB while testing DNA with very low concentration of mutant genome.All the designed assays were also strongly approved for linearity using mixtures of mutated and WT DNA from cell lines (0.01% to 100% mutated allele frequency). Specificity, linearity and sensibility of the selected assays were validated on genomic DNA. Later on, we investigate genomic DNA of 3 MDS patients with RS and harboring JAK2 and SF3B1 mutations. For these patients, we obtained comparable results using both NGS and picoliter-droplet digital PCR in term of mutant allele burden quantification. Moreover, a triplex assay allowing mutant allele discrimination in JAK2 and SF3B1 genes was established on these patients.Further analyses were conducted on living cells harboring JAK2 or NRAS mutations. This approach was first conducted using a "home made" microfluidic system based on the detection of fluorescent probes in living cells encapsulated into agarose beeds. We obtained specific fluorescent signals corresponding to the genotypes. In parallel, an alternative method based on the QX100™Droplet Digital™PCR system (Biorad) also demonstrated the feasibility of allelic discrimination in living cells. Experiments based on frozen cells of MDS patients are currently under investigation.ConclusionThis study is the first application of multi-target digital PCR used to detect and quantify somatic mutations recurrently found in MDS. Analyses of the clonal architecture determined on living cells and its evolution upon treatment in MDS patients with RS by this approach will help us to investigate the monitoring of the therapeutic response. Our study supports a proof of principle for further large-scale analyses of MDS patients at diagnosis and follow-up. DisclosuresNo relevant conflicts of interest to declare.

Choosing Thresholds for Statistical Signal Detection with the Proportional Reporting Ratio

Identification of potential drug safety problems using statistical screening algorithms in routinely collected databases of adverse drug reactions (ADRs) requires decision rules based on thresholds of the chosen parameters. Choosing higher or lower thresholds changes both the sensitivity of the screening and the number of false alarms produced, and thus has an impact on the effectiveness of the detection process. The aim of this study was to evaluate the impact on the effectiveness of signal detection activities of choosing different warning thresholds for the proportional reporting ratio (PRR) and for the count of reports of any drug-event combination. Signal detection methods were tested within the EudraVigilance database of suspected ADRs. Using an established set of known ADRs, the number that could be detected and the changes in time gained for earlier investigation of the signal were calculated over a range of signal detection thresholds. These figures were set against the number of false positive signals produced by the statistical signal detection algorithms. Higher thresholds for the lower confidence bound of the PRR produced fewer false positives but this benefit was offset by important losses of sensitivity in the detection of ADRs. By contrast, increases in the threshold for the count of a specific drug-event combination produced fewer false positives with little loss of either sensitivity or time gained for investigation of adverse events. A threshold of five compared with the current European Medicines Agency threshold of three gave a reduction of 25 % in false positive signals in return for a loss of 12 % in true signals detected early. Changes in the standard threshold for the count of drug-event combinations can result in a substantial improvement in efficiency of the signal detection process. Initially this change might be applied only to products with a well-established safety profile.

Seasonal dynamics of harmful algae in outer Oslofjorden monitored by microarray, qPCR, and microscopy

Monitoring of marine microalgae is important to predict and manage harmful algal blooms. Microarray Detection of Toxic ALgae (MIDTAL) is an FP7-funded EU project aiming to establish a multi-species microarray as a tool to aid monitoring agencies. We tested the suitability of different prototype versions of the MIDTAL microarray for the monthly monitoring of a sampling station in outer Oslofjorden during a 1-year period. Microarray data from two different versions of the MIDTAL chip were compared to results from cell counts (several species) and quantitative real-time PCR (qPCR; only Pseudochattonella spp.). While results from generation 2.5 microarrays exhibited a high number of false positive signals, generation 3.3 microarray data generally correlated with microscopy and qPCR data, with three important limitations: (1) Pseudo-nitzschia cells were not reliably detected, possibly because cells were not sufficiently retained during filtration or lysed during the extraction, and because of low sensitivity of the probes; (2) in the case of samples with high concentrations of non-target species, the sensitivity of the arrays was decreased; (3) one occurrence of Alexandrium pseudogonyaulax was not detected due to a 1-bp mismatch with the genus probe represented on the microarray. In spite of these shortcomings our data demonstrate the overall progress made and the potential of the MIDTAL array. The case of Pseudochattonella - where two morphologically similar species impossible to separate by light microscopy were distinguished - in particular, underlines the added value of molecular methods such as microarrays in routine phytoplankton monitoring.

Beware of risk for increased false positive rates in genome-wide association studies for phenotypic variability

Beware of risk for increased false positive rates in genome-wide association studies for phenotypic variability

Power and robustness of three whole genome association mapping approaches in selected populations

Selection is known to influence the linkage disequilibrium (LD) pattern in livestock populations. Spurious LD may lead to a higher number of false-positive signals in whole genome association mapping experiments. We compared three approaches for whole genome association mapping in a simulation study: single marker regression (SMR), a two-step approach, which analyses residuals corrected for family effects with an SMR (GRAMMAR), and a combined linkage and LD approach, which applies the quantitative transmission disequilibrium test to the Mendelian sampling term (MTDT). Three different scenarios were simulated: idealized random mating, limited number of parents and directional selection. The number of false-positive associations increased when the number of parents was limited. Mapping accuracy was the worst in the scenario with directional selection for all approaches. As SMR produced a high number of false-positive signals in small populations, results of whole genome scans in livestock analysed with SMR should be considered with caution. GRAMMAR was the most accurate approach, but also the least powerful one. The Bonferroni-corrected significance threshold seemed to be too stringent for this approach. Results obtained with MTDT changed only slightly with selected populations. MTDT combined sufficient power with a manageable number of false-positive associations in all scenarios.

Statistical methods for examining genetic influences of resistance to anti-epileptic drugs

It is generally accepted that genetics may be an important factor in explaining the variation between patients’ responses to certain drugs. However, identification and confirmation of the responsible genetic variants is proving to be a challenge in many cases. A number of difficulties that may be encountered in pursuit of these variants, such as nonreplication of a true effect, population structure and selection bias, can be mitigated or at least reduced by appropriate statistical methodology. Another major statistical challenge facing pharmacogenetic studies is the detection of possible small polygenic effects using large volumes of genetic data, while controlling the number of false-positive signals. Here, we review statistical design and analysis options available for investigations of genetic resistance to anti-epileptic drugs.

Evaluation of gene expression analysis using RNA-targeted partial genome arrays

Highly parallel cDNA targeting microarrays have been established over the last years as the quasi-standard for genome wide expression profiling in pro- and eukaryotes. Protocols for the direct detection of RNA or aRNA (amplified RNA) are currently emerging. This allows to circumvent the bias introduced by enzymatic target molecule preparation. To systematically evaluate the extent of non-specific target binding on oligonucleotide microarrays designed for total RNA expression profiling, a model system of 70-mer probes targeting genes involved in magnetosome formation ( mam genes) of the bacterium Magnetospirillum gryphiswaldense was established utilizing wild-type strain MSR-1 and an isogenic deletion mutant MSR-1B that lacks all mam genes. An optimized protocol for the direct chemical labelling of total cellular RNAs was used. A linear correlation between the amount of applied RNA and the mean global background intensity was found which enables a simple and unbiased way of normalizing the data. The results obtained with the mam deletion mutant MSR-1B revealed a significant number of false positive signals, even under optimal hybridization conditions. This indicates a high degree of non-specific binding in microarray experiments when using longer oligo- or polynucleotides and RNA as target molecule. Comparative microarray analysis of an MSR-1B culture and two MSR-1 wild-type cultures grown under different conditions was done via a three-colour hybridization assay. The additional information provided by the MSR-1B transcriptome revealed differential gene expression in the two MSR-1 cultures, which was otherwise undetectable.

Identification of probable genotyping errors by consideration of haplotypes

Undetected genotyping errors pose a problem in genetic epidemiological studies, as they may invalidate statistical analysis or reduce its power. Haplotype analysis requires an improved standard of the data, because a haplotype can be inferred correctly only if the genotypes of all its markers are correct. Here, we present a method that identifies probable genotyping errors in trio samples with the help of the estimated haplotype frequency distribution of the sample. If the likelihood of the most likely haplotype explanation depends strongly on just one genotype, in the sense that setting the genotype to be missing leads to a much more likely haplotype explanation, this genotype is considered as a potential genotyping error. We describe a method that systematically searches the whole data set for such potential errors. Based on the haplotype distribution of a real data set, we carry out a simulation study to estimate the sensitivity and specificity of the method. In addition, we apply our approach to the real data set itself. Potentially erroneous genotypes are re-determined via sequencing. The results of both the simulation study and of the application to the real data set show that a considerable proportion of true genotyping errors is detected and that the number of false-positive signals is acceptable. We conclude that it is indeed possible to identify probable genotyping errors by considering haplotypes. The method described here will be part of the next release of our FAMHAP software.

Acquisition of fluorescence and reflectance spectra during routine bronchoscopy examinations using the ClearVu Elite™ device: Pilot study

Fluorescence flexible bronchoscopy (FFB) has proved to be very useful for detecting carcinoma in situ (CIS) and pre-cancerous lesions of the lung that are generally occult to white light reflectance bronchoscopy (WLRB). However, the increased sensitivity has caused a significant decrease of specificity, resulting in a large number of false positive signals that lead to a significant number of unnecessary biopsies. We have been planning to test a hypothesis that reflectance spectra and fluorescence spectra could be used to distinguish the true positive lesions from the false positive ones. To properly test such hypothesis, several thousand patients will need to be examined to obtain sufficient data from different lung lesions. Towards this goal, we have developed and have been testing a special system (ClearVu Elite ) that facilitates acquisition of both WLRB and FFB spectra during routine bronchoscopy examinations. In this pilot study we examined (1) if such could be used in a practical, routine clinical conditions without affecting commonly used bronchoscopy procedures; (2) if the spectral data obtained from the images are identical to those obtained with fiber-optic probes; and (3) if the few malignant and early neoplastic lesions available for this pilot study show any differences from normal lung tissue. The study population consisted of 63 patients that were suspicious for presence of lung cancer (19 female and 44 male, smokers and non-smokers). All were examined by diagnostic bronchoscopy using ClearVu Elite during a period of 8 weeks. The experimental system provides excellent real time WLRB and FFB that can be switched from one to another mode instantaneously. Acquisition of both white light reflectance spectra and fluorescence spectra is acquired in real time through the images and does not prolong the routine WLRB and FFB procedures. The experimental bronchoscopy procedure is as simple as conventional bronchoscopy, adding on average less than 5 min to the 20 min procedure. The acquired spectra are identical to those obtained by fiber-optic probes. In all of the limited number of malignant and early neoplastic lesions, there were differences found which are sufficiently pronounced to warrant initiation of a large, multicenter study for development of differentiating algorithms of statistical validity.

Artefactual increasing frequency of omphaloceles in the Northern Netherlands: lessons for systematic analysis of apparent epidemics.

While monitoring birth defects in a registry, statistically significant increases in prevalence occasionally occur. In the European Registration Of Congenital Anomalies (EUROCAT) in the Northern Netherlands 20000 births are monitored every year. For omphaloceles, a steady increase in the prevalence from 0.86 per 10000 live- and stillbirths in 1981-1983 to 3.11 per 10000 live- and stillbirths in 1994 was seen in the three northern provinces of The Netherlands. A stepwise enquiry into this increase, which included checking for misclassification and change in coding and ascertainment when necessary, was done. All cases of omphalocele and associated or similar birth defects registered at the EUROCAT registry were retrieved and if necessary recoded. This study showed that the increase reported previously was not a true time trend. A few cases of e.g. diastasis recti and trisomy 18 were misclassified. The prevalence in more recent years is comparable with that in the rest of Europe, whereas it used to be lower. There was an increase in isolated omphalocele, but the numbers are small. The stepwise enquiry described should be a standard procedure after noticing an increasing prevalence in a registry. A better subdivision, e.g. in isolated cases versus children with multiple congenital anomalies, before monitoring can contribute to a lower number of false positive signals.