Discrepant Resolution Research Articles

Clinical evaluation of the APAS® Independence: Automated imaging and interpretation of urine cultures using artificial intelligence with composite reference standard discrepant resolution

BackgroundThis study reports the outcome of the first evaluation of the APAS® Independence for automated reading and preliminary interpretation of urine cultures in the routine clinical microbiology laboratory. In a 2-stage evaluation involving 3000 urine samples, two objectives were assessed; 1) the sensitivity and specificity of the APAS® Independence compared to microbiologists using colony enumeration as the primary determinant, and 2) the variability between microbiologists in enumerating bacterial cultures using traditional culture reading techniques, performed independently to APAS® Independence interpretation. MethodsRoutine urine samples received into the laboratory were processed and culture plates were interpreted by standard methodology and with the APAS® Independence. Results were compared using typical discrepant result resolution and with a composite reference standard, which provided an alternative assessment of performance. ResultsThe significant growth sensitivity of the APAS® Independence was determined to be 0.919 with a 95% confidence interval of (0.879, 0.948), and the growth specificity was 0.877 with a 95% confidence interval of (0.827, 0.916). Variability between microbiologists was demonstrated with microbiologist bi-plate enumerations in agreement with the consensus 88.6% of the time. ConclusionThe APAS® Independence appears to offer microbiology laboratories a mechanism to standardise the processing and assessment of urine cultures whilst augmenting the skills of specialist microbiology staff.

2564. Clinical Validation of a Commercial LAMP Test for Ruling Out Malaria in Returning Travelers: A Prospective Diagnostic Trial

BackgroundThe mainstay of malaria diagnosis relies on rapid diagnostic tests (RDT) and Giemsa-stained microscopy both of which lack analytical sensitivity. This leads to repeat testing to rule out malaria. Nucleic acid amplification (NAT) methods are more sensitive, but testing requires technical proficiency beyond the average clinical laboratory.MethodsWe conducted a prospective diagnostic trial of the Meridian illumigene Malaria assay (LAMP) compared with reference microscopy and RDT (BinaxNOW Malaria) in returning travelers in Western Canada between June 2017 and January 2018. Returning travelers with signs and symptoms of fever were enrolled into the study. RDT, microscopy, and LAMP assays were performed simultaneously. To increase the yield of positive specimens for all species of malaria, retrospective specimens of Plasmodium vivax, P. ovale, and P. malariae species were supplemented. Real-time (RT)–PCR testing was performed on all specimens to resolve discrepancies. A cost–benefit analysis was performed.ResultsA total of 296 consecutive patients (50.7% male, mean age 32.5) were enrolled, most visiting friends and relatives (43.2%), traveling to Asia (48.4%), presenting with fever (88.9%), not taking prophylaxis (82.8%), and treated as outpatients (84.3%). In the prospective arm, LAMP had a sensitivity of 98.1% (95% CI 89.9–99.9) and a specificity of 97.6% (95% CI 95.2–99.0) versus microscopy. After discrepant resolution with RTPCR, LAMP had a sensitivity of 100% (95% CI 93.9–100) and a specificity of 100% (95% CI 98.7–100) versus microscopy. When including retrospective specimens, LAMP had a sensitivity of 98.7% (95% CI 92.7–99.9) and a specificity of 97.6% (95% CI 95.2–99.1) versus microscopy, and after discrepant resolution of this set, LAMP had a sensitivity of 100% (95% CI 95.5–100) and a specificity of 100% (95% CI 98.7–100). The rate of invalid tests with LAMP was 3.05%. After discrepant resolution, RDT had a sensitivity of 83.3% (95% CI 58.6–96.4) and a specificity of 96.2% (95% CI 93.2–98.1) versus microscopy. A cost–benefit analysis of reagents and labor suggests up to USD 13 savings per specimen using a revised algorithm with LAMP screening.ConclusionA novel, highly sensitive testing algorithm for malaria screening with associated cost savings in the nonendemic setting is proposed.Disclosures D. Pillai, Meridian Biosciences: None, Diagnostic testing material for study.

Clinical Validation of a Commercial LAMP Test for Ruling out Malaria in Returning Travelers: A Prospective Diagnostic Trial.

The mainstay of malaria diagnosis relies on rapid diagnostic tests (RDTs) and microscopy, both of which lack analytical sensitivity. This leads to repeat testing to rule out malaria. A prospective diagnostic trial of the Meridian illumigene Malaria assay (loop-mediated isothermal amplification [LAMP]) was conducted comparing it with reference microscopy and RDTs (BinaxNOW Malaria) in returning travelers between June 2017 and January 2018. Returning travelers with signs and symptoms of malaria were enrolled in the study. RDTs, microscopy, and LAMP assays were performed simultaneously. A total of 298 patients (50.7% male; mean age, 32.5 years) were enrolled, most visiting friends and relatives (43.3%), presenting with fever (88.9%), not taking prophylaxis (82.9%), and treated as outpatients (84.1%). In the prospective arm (n = 348), LAMP had a sensitivity of 98.1% (95% confidence interval [CI], 90.0%–100%) and a specificity of 97.6% (95% CI, 95.2%–99.1%) vs microscopy. After discrepant resolution with real-time polymerase chain reaction, LAMP had a sensitivity of 100% (95% CI, 93.7%–100%) and a specificity of 100% (95% CI, 98.7%–100%) vs microscopy. After discrepant resolution, RDTs had a sensitivity of 83.3% (95% CI, 58.6%–96.4%) and a specificity of 96.2% (95% CI, 93.2%–98.1%) vs microscopy. When including retrospective specimens (n = 377), LAMP had a sensitivity of 98.8% (95% CI, 93.2%–100%) and a specificity of 97.6% (95% CI, 95.2%–99.1%) vs microscopy, and after discrepant resolution of this set, LAMP had a sensitivity of 100% (95% CI, 95.8%–100%) and a specificity of 100% (95% CI, 98.7%–100%). A cost-benefit analysis of reagents and labor suggests savings of up to USD$13 per specimen using a novel algorithm with LAMP screening.

Validation of midsagittal jaw movements to measure sleep in healthy adults by comparison with actigraphy and polysomnography.

OBJECTIVE In a device based on midsagittal jaw movements analysis, we assessed a sleep-wake automatic detector as an objective method to measure sleep in healthy adults by comparison with wrist actigraphy against polysomnography (PSG). METHODS Simultaneous and synchronized in-lab PSG, wrist actigraphy and jaw movements were carried out in 38 healthy participants. Epoch by epoch analysis was realized to assess the ability to sleep-wake distinction. Sleep parameters as measured by the three devices were compared. This included three regularly reported parameters: total sleep time, sleep onset latency, and wake after sleep onset. Also, two supplementary parameters, wake during sleep period and latency time, were added to measure quiet wakefulness state.RESULTS The jaw movements showed sensitivity level equal to actigraphy 96% and higher specificity level (64% and 48% respectively). The level of agreement between the two devices was high (87%). The analysis of their disagreement by discrepant resolution analysis used PSG as resolver revealed that jaw movements was right (58.9%) more often than actigraphy (41%). In sleep parameters comparison, the coefficient correlation of jaw movements was higher than actigraphy in all parameters. Moreover, its ability to distinct sleep-wake state allowed for a more effective estimation of the parameters that measured the quiet wakefulness state.CONCLUSIONS Midsagittal jaw movements analysis is a reliable method to measure sleep. In healthy adults, this device proved to be superior to actigraphy in terms of estimation of all sleep parameters and distinction of sleep-wake status.

Serologic Testing for Syphilis: Benefits and Challenges of a Reverse Algorithm

Syphilis is a human infection of global importance. Its diagnosis can be challenging, requiring construction of a serologic profile based on the results of at least two types of antibody tests: treponemal and nontreponemal. The traditional approach to the serodiagnosis of syphilis has been the use of a nontreponemal screening assay followed by the performance of a treponemal confirmatory test if the initial nontreponemal screening test was reactive. With the increasing availability of automated, easier-to-perform, and rapid treponemal assays, an increasing number of laboratory testing sites are adopting reverse sequence screening for the serodiagnosis of syphilis: screening with a treponemal assay first, then confirmation with a nontreponemal assay and, when necessary, discrepant resolution using another treponemal test. In addition to offering automation and increased throughput, a reverse algorithm can increase disease detection, especially in late latent and early primary stages of infection when the nontreponemal antibody test may be nonreactive. However, a disadvantage to this approach is that there can be an increase in false-positive test results. This article reviews the clinical and workflow benefits and limitations of a reverse testing algorithm and discusses current guidance available from the Centers for Disease Control and Prevention.

Evaluation of the Hologic Gen-Probe PANTHER, APTIMA Combo 2 Assay in a Tertiary Care Teaching Hospital

To evaluate the performance of the Hologic Gen-Probe (San Diego, CA) PANTHER system. The performance of PANTHER was compared with the Hologic Gen-Probe TIGRIS and/or Roche (Indianapolis, IN) COBAS AMPLICOR systems through testing of patient specimens and the spiked-urine matrix. After discrepant resolution, PANTHER demonstrated a 99.3% (95% confidence interval [CI], 96.0%-99.9%) positive and 100% (98.5%-100.0%) negative agreement for Chlamydia trachomatis (CT) and 100% (96.6%-100.0%) positive and 100% (98.6%-100.0%) negative agreement for Neisseria gonorrhoeae (NG) for all male, female, unsexed, and NG-spiked female urine specimens combined. For other specimen types collectively, the PANTHER demonstrated 100% (95% CI, 90.6%-100.0%) positive and 100% (88.3%-100.0%) negative agreement for CT and 90.9% (62.8%-98.4%) positive and 100% (93.5%-100.0%) negative agreement for NG. Analytical sensitivity of the PANTHER in urine matrix was similar to the TIGRIS system. The PANTHER system provides an excellent new addition to options for detecting CT and NG, is appropriate for testing urine samples, and will facilitate high-throughput testing in the clinical laboratory.

Altın Standart Testin Olmadığı Durumda Tanı Testlerinin Performanslarının Değerlendirilmesi

Situations are encountered that gold standard doesn’t exist and two or more imprecise referencetests are available in the diagnosis of a specific disease clinically. There are three basicapproaches including Discrepant Resolution (DR), Composite Reference Standard (CRS) andLatent Class model (LCA) in literature for assessing the performance of diagnostic tests. Thisstudy is aimed to evaluate the three basic approaches. LCA has a more complex structure thanDR and CRS approach. LCA is more advantageous than CRS and DR methods without anyassumptions about resolver test or other imperfect reference tests specify. As a result, LCAmethods which are developed for assessing the performance of new diagnostic test and reducingthe bias because of imperfect reference tests give more reliable results than other methods

With or Without a Gold Standard

To explore causal relationships between exposure and outcome, epidemiologists must rely on accurate measurements of both. Misclassification of either exposure or outcome will obscure causality, ie, by an inability to distinguish exposed from unexposed or “diseased” from “nondiseased.” Whenever new and purportedly better (but nonvalidated) measurements of exposure or outcome become available, epidemiologists are faced with a fundamental question of “Which test is better?” Tools for answering this question differ depending on whether a true gold standard is available. In this issue of EPIDEMIOLOGY, articles by Pepe 1 and Hagdu 2 address some of the issues surrounding comparisons of diagnostic tests with 1 and without 2 a gold standard. The receiver-operator characteristics curve (ROC) is a popular summary measure of the discriminatory ability of a clinical marker that can be used when there is a gold standard. The ROC plots sensitivity against 1-specificity (true versus false positivity) for all thresholds that could have been used to define “test positive.” An ROC curve is assessed by measuring the area under the curve (AUC), which ranges from 0.5 (no discriminatory ability) to 1.0 (perfect discriminatory ability). Two diagnostic tests can be compared by calculating the difference between the areas under their 2 ROC curves. Pepe 1 offers an alternative (nonparametric) presentation of ROC analysis that may make such analysis more accessible to epidemiologic researchers. Pepe relates the ROC curve to standardized values of the clinical marker, termed “placement values.” The placement value for an affected individual with maker value Y is the proportion of unaffected subjects (the reference population) with values higher than Y. To extend this idea to the whole population, the mean placement value or average of the percentiles is computed by averaging the placement values of the affected subjects, with the mean placement value reflecting the distribution of values in the control population. A low mean placement value indicates that the clinical marker distinguishes patients with an outcome from those without an outcome in that population. Using the fact that the ROC curve is the probability distribution of the placement values, Pepe demonstrates that the mean placement value is equal to 1-AUC. A nice feature of using placement values is that they are amenable to regression analysis to identify and control for factors that modify the performance of the clinical measurement. Both of these related approaches use all thresholds for the definition of “test positive.” In practice, users often confine their attention to regions of the ROC curve that correspond to the most clinically relevant values of test sensitivity or specificity. For example, a test that performs well for low specificity but poorly for high specificity may not be desirable. In this case, the partial area under the curve is a more appropriate measure of test performance than the AUC.

Reporting results from studies evaluating diagnostic tests

Evaluating a new diagnostic test requires careful planning. It involves choosing the appropriate comparative procedure, patients, specimens, and individuals performing the tests. The type of study design used to evaluate a new test has a direct impact on how the study results can be reported. This article describes some statistically appropriate and inappropriate practices for reporting results from different studies evaluating qualitative diagnostic tests. Special attention is given to describing a practice called discrepant resolution and its associated problems.

Some issues in resolution of diagnostic tests using an imperfect gold standard.

As a subject's true disease status is seldom known with certainty, it is necessary to compare the performance of new diagnostic tests with those of a currently accepted but imperfect 'gold standard'. Errors made by the gold standard mean that the sensitivity and specificity calculated for the new test are biased, and do not correctly estimate the new method's sensitivity and specificity. The traditional approach to this problem was 'discrepant resolution', in which the subjects for whom the two methods disagreed were subjected to a third 'resolver' test. Recent work has pointed out that this does not automatically solve the problem. A sounder approach goes beyond the discordant test results and tests at least some of the subjects with concordant results with the resolver also. This leaves some issues unresolved. One is the basic question of the direction of biases in various estimators. We point out that this question does not have a simple universal answer. Another issue, if one is to test a sample of the subjects with concordant results rather than all cases, is how to compute estimates and standard errors of the measures of test performance, notably sensitivity and specificity of the test method relative to the resolver. Expressions for these standard errors are given and illustrated with a numeric example. It is shown that using just a sample of subjects with concordant results may lead to great savings in assays. The design issue of how many concordant cells to test depends on the numbers of concordants and discordants. The formulae given show how to evaluate impact of different choices for these numbers and hence settle on a design that gives the required precision of estimates.

Comparison of Gen-Probe AccuProbe Group B streptococcus culture identification test with conventional culture for the detection of Group B streptococci in broth cultures of vaginal-anorectal specimens from pregnant women☆

The performance of the AccuProbe Group B Streptococcus Culture Identification Test (Gen-Probe Incorporated, San Diego, CA, USA) for the detection of group B streptococci (GBS) directly from LIM broth cultures of vaginal-anorectal swab specimens from pregnant women (two swabs per patient in most cases) was evaluated by comparing results to those of conventional GBS culture. Of 411 specimens analyzed, 82 were positive and 312 were negative for GBS by both methods. After initial testing, the percent agreement was 95.9%. The initial sensitivity, specificity, and positive and negative predictive values for the AccuProbe test were 90.1%, 97.5%, 91.1%, and 97.2%, respectively. Results were discrepant for 17 specimens: eight were GBS positive by probe and negative by culture; nine were negative by probe and positive by culture. To resolve discrepancies, culture plates were re-examined for GBS colonies, AccuProbe testing was repeated on the initial LIM broth cultures, and the second swab (if received) was inoculated to LIM broth for AccuProbe testing after overnight incubation. After discrepant resolution testing, the percent agreement between the two test methods was 97.8%. The final sensitivity, specificity, and positive and negative predictive values for the AccuProbe test were 95.6%, 98.4%, 94.6%, and 98.7%, respectively. These data suggest that the AccuProbe test is a reliable method for detecting GBS in vaginal-anorectal specimens, providing results more rapidly than conventional culture. However, strict adherence to the manufacturer’s test protocol is necessary to limit technical errors.