Spectrum Bias Research Articles

O-192 Kisspeptin as a test of hypothalamic function in women presenting with oligo-amenorrhea

Abstract Study question Can kisspeptin be used as a novel tool to interrogate hypothalamic function in women presenting with oligo-amenorrhea? Summary answer The hormonal response to kisspeptin offers a novel approach for the assessment of hypothalamic function in patients presenting with oligo-amenorrhea of different aetiologies. What is known already Polycystic ovary syndrome (PCOS) and functional hypothalamic amenorrhea (FHA) are the two commonest causes of anovulation causing oligo-amenorrhea in pre-menopausal women. Additionally, congenital (CHH) and functional hypogonadotropic hypogonadism can present with similar endocrine profiles. Importantly, these conditions are caused and characterised by abnormal hypothalamic function. For instance, gonadotropin-releasing hormone (GnRH) pulsatility is increased in PCOS but reduced in FHA. Despite differences in pathophysiology, differentiating these diagnoses can be challenging in clinical practice. Kisspeptin is known to stimulate hypothalamic GnRH neurons. Thus, kisspeptin could represent a novel tool to interrogate hypothalamic function in women presenting with reproductive disorders causing oligo-amenorrhea. Study design, size, duration Single-centre, prospective, experimental study of healthy women (n = 33), women with PCOS (n = 30), FHA (n = 30), or CHH (n = 8), aged 18-35 years and not taking hormonal treatment. Participants were recruited for clinical studies between September 2019 and September 2023 and underwent two separate study visits. The visits assessed the gonadotropin response to hypothalamic stimulation with an intravenous bolus of kisspeptin-54 (9.6nmol/kg) and the pituitary response to an intravenous bolus of GnRH (100 mcg). Participants/materials, setting, methods All participants took part in two study visits. Serum luteinizing hormone (LH), and follicle stimulating hormone (FSH) levels were measured every 15 minutes for 8 hours after administration of kisspeptin-54 and GnRH. The maximal rise in LH and FSH from baseline was compared between groups using an unpaired t-test. Receiver operator characteristic (ROC) analysis was used to assess the discriminatory potential of the hormone response to kisspeptin and GnRH. Main results and the role of chance The gonadotropin response to kisspeptin was increased in women with FHA compared to all other groups. The mean (SD) of the maximal rise in LH (IU/L) after kisspeptin was 9.7 (11.4) in healthy women, 9.7 (8.3) in women with PCOS, 17.6 (10.8) in women with FHA, and 1.0 (1.1) in women with CHH. The mean (SD) of the maximal rise in FSH (IU/L) after kisspeptin was 4.2 (3.4) in healthy women, 2.8 (2.1) in women with PCOS, 9.2 (5.6) in women with FHA, and 0.7 (0.5) in women with CHH. In lean women (BMI <25 kg/m2), the maximal rise in FSH after kisspeptin differentiated women with PCOS from those with FHA (area under ROC curve 0.91, p = 0.0001). Furthermore, the maximal rise in FSH after kisspeptin differentiated women with FHA from women with CHH (auROC 1.00, p < 0.0001), whereas the equivalent auROC for FSH rise after GnRH was 0.7 (p = 0.086). Limitations, reasons for caution Participants included in this study had clearcut diagnoses of PCOS, FHA, or CHH prior to inclusion, however this could increase the risk of spectrum bias. Whilst this study provides mechanistic insight into hypothalamic dysfunction in these conditions, further prospective validation of discriminatory thresholds in different settings is required. Wider implications of the findings The specific action of kisspeptin at hypothalamic GnRH neurons enables its use to assess hypothalamic function in patients with oligo-amenorrhea due to different anovulatory disorders. Our data suggests that the hormonal response to kisspeptin has promising diagnostic potential to aid in the evaluation of women with oligo-amenorrhea. Trial registration number ISRCTN21681316

Risk of Bias Assessment and Risk Minimisation Strategies in COVID-19 Diagnostic Test Accuracy Study

It is paramount to assess the risk of biases in may arise from diagnostic test accuracy (DTA) study as it will affect the accuracy and validity of the tests. These biases can be found in published researches and here we look at COVID-19 DTA studies. The evaluation of bias risk in diagnostic research is mainly performed using QUADAS-2. The aim of this review was to determine potential selection and information biases in diagnostic test accuracy studies and strategies to minimize risk of biases. Literature review related to diagnostic test accuracy study is identified through an online search of databases namely PubMed, ScienceDirect, Research Gate, Google Scholar, and official government websites range. Six potential biases in four QUADAS-2 domains are identified in COVID-19 diagnostic test accuracy study which are 1) spectrum bias in patient selection; 2) interpretation bias in index test; 3) differential misclassification bias and nondifferential misclassification bias in reference standard; and 4) partial verification bias and differential verification bias in patient flow. The identified biases exert effects on accuracy of COVID-19 diagnostic tests. Six strategies are recommended to reduce these biases, hence, improving the accuracy of COVID-19 diagnostic tests. The best diagnostic test can give benefits to the population in the mass screening program during COVID-19.

SARS-CoV-2 Serology Across Scales: A Framework for Unbiased Estimation of Cumulative Incidence Incorporating Antibody Kinetics and Epidemic Recency.

Serosurveys are a key resource for measuring severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) population exposure. A growing body of evidence suggests that asymptomatic and mild infections (together making up over 95% of all infections) are associated with lower antibody titers than severe infections. Antibody levels also peak a few weeks after infection and decay gradually. We developed a statistical approach to produce estimates of cumulative incidence from raw seroprevalence survey results that account for these sources of spectrum bias. We incorporate data on antibody responses on multiple assays from a postinfection longitudinal cohort, along with epidemic time series to account for the timing of a serosurvey relative to how recently individuals may have been infected. We applied this method to produce estimates of cumulative incidence from 5 large-scale SARS-CoV-2 serosurveys across different settings and study designs. We identified substantial differences between raw seroprevalence and cumulative incidence of over 2-fold in the results of some surveys, and we provide a tool for practitioners to generate cumulative incidence estimates with preset or custom parameter values. While unprecedented efforts have been launched to generate SARS-CoV-2 seroprevalence estimates over this past year, interpretation of results from these studies requires properly accounting for both population-level epidemiologic context and individual-level immune dynamics.

Fault Damage Zone Effects on Ground Motions during the 2019 Mw 7.1 Ridgecrest, California, Earthquake

ABSTRACT We have simulated 0–3 Hz deterministic wave propagation in the Southern California Earthquake Center Community Velocity Model (CVM) version CVM-S4.26-M01 for the 2019 Mw 7.1 Ridgecrest earthquake. A data-constrained high-resolution fault zone model (Zhou et al., 2022) is incorporated into the CVM to investigate the effects of the near-fault low-velocity zone (LVZ) on the resulting ground motions, constrained by strong-motion data recorded at 161 stations. The finite-fault source used for the simulation of the Ridgecrest event was obtained from the Liu et al. (2019) kinematic inversion, enriched by noise following a von Karman correlation function above ∼1 Hz with a f−2 high-frequency decay. Our results show that the heterogeneous near-fault LVZ inherent to the fault zone structure significantly perturbs the predicted wave field in the near-source region, in particular by more accurately generating Love waves at its boundaries. The fault zone decreases the 0.1–0.5 Hz mean absolute Fourier amplitude spectrum bias to seismic recordings for all sites in the model and in the Los Angeles basin area (∼200 km from the source) by 16% and 26%, respectively. The fault zone structure generally improves modeling of the long-period features in the data and lengthens the coda-wave trains, in better agreement with observations. The favorable fit to data was obtained with a model including high-resolution surface topography, a 700-m-thick geotechnical layer and frequency-dependent anelastic attenuation in the model domain, with QS=0.1VS and QS(f)=0.1VSf0.5 (VS in m/s) for frequencies lower and higher than 1 Hz, respectively. We recommend that a data-constrained fault zone velocity structure, where available, be included in ground-motion modeling to obtain the least-biased fit to observed seismic data.

Insufficient evidence for the validity of the Language Development Survey and the MacArthur-Bates Communicative Development Inventories as screening tools: A critical review.

The Language Development Survey (LDS) and the MacArthur-Bates Communicative Development Inventories (MB-CDI) are two parental report forms that have been productive in providing data on early child language during the past 30 years. The instruments have been used both in studies relating to typical developing children and in screening for language difficulties. To review the evidence for the LDS and the MB-CDI utility as screening instruments. A literature search in PubMed and PsychInfo identified 16 articles based on LDS or MB-CDI that reported statistics pertinent to early screening for language difficulties. It was found that most reviewed studies were explorative in nature and tried out different versions of the screening test, including different cut-off values, multiple reference tests, small sample sizes and rarely reported confidence intervals. Spectrum, verification and review biases were common. Moreover, no study could convincingly show that the actual diagnostic accuracy was sufficient for clinical use. There is insufficient evidence that the LDS or any version of the MB-CDI is a valid tool for screening for language difficulties. Of course, this is not to say that sufficient evidence will not be achieved in future studies, or that the instruments do not work well for purposes other than screening. What is already known on this subject The LDS and the MB-CDI are two often-used instruments assessing various aspects of early child language by parental reports. Both instruments have also been used in screening for early language difficulties. What this study adds This study reveals that most published studies in which the classification accuracy of LDS and the MB-CDI has been investigated contain serious methodological shortcomings limiting conclusions about their validity. Currently, there is no good evidence about the usefulness of the LDS and the MB-CDI as general screening tools for language difficulties. What are the potential or actual clinical implications of this work? The LDS and MB-CDI should not be used as screening tools for language difficulties until better evidence of their effectiveness has been demonstrated.

Developing a reference standard for pertussis by applying a stratified sampling strategy to electronic medical record data

Pertussis surveillance remains essential in Canada, but ascertainment bias limits the accuracy of surveillance data. Introducing other sources to improve detection has highlighted the importance of validation. However, challenges arise due to low prevalence, and oversampling suspected cases can introduce partial verification bias. The aim of this study was to build a reference standard for pertussis validation studies that provides adequate analytic precision and minimizes bias. We used a stratified strategy to sample the reference standard from a primary care electronic medical record cohort. We incorporated abstractor notes into definite, possible, ruled-out, and no mention of pertussis classifications which were based on surveillance case definitions. We abstracted eight hundred records from the cohort of 404,922. There were 208 (26%) definite and 261 (32.6%) possible prevalent pertussis cases. Classifications demonstrated a wide variety of case severities. Abstraction reliability was moderate to substantial based on Cohen's kappa and raw percent agreement. When conducting validation studies for pertussis and other low prevalence diseases, this stratified sampling strategy can be used to develop a reference standard using limited resources. This approach mitigates verification and spectrum bias while providing sufficient precision and incorporating a range of case severities.

External Validation of a Multivariable Prediction Model for Placenta Accreta Spectrum.

Placenta accreta spectrum (PAS) is a disorder of abnormal placentation associated with severe postpartum hemorrhage, maternal morbidity, and mortality. Predelivery prediction of this condition is important to determine appropriate delivery location and multidisciplinary planning for operative management. This study aimed to validate a prediction model for PAS developed by Weiniger et al in 2 cohorts who delivered at 2 different United States tertiary centers. Cohort A (Brigham and Women's Hospital; N = 253) included patients with risk factors (prior cesarean delivery and placenta previa) and/or ultrasound features of PAS presenting to a tertiary-care hospital. Cohort B (Columbia University Irving Medical Center; N = 99) consisted of patients referred to a tertiary-care hospital specifically because of ultrasound features of PAS. Using the outcome variable of surgical and/or pathological diagnosis of PAS, discrimination (via c-statistic), calibration (via intercept, slope, and flexible calibration curve), and clinical usefulness (via decision curve analysis) were determined. The model c-statistics in cohorts A and B were 0.728 (95% confidence interval [CI], 0.662-0.794) and 0.866 (95% CI, 0.754-0.977) signifying acceptable and excellent discrimination, respectively. The calibration intercept (0.537 [95% CI, 0.154-0.980] for cohort A and 3.001 [95% CI, 1.899- 4.335] for B), slopes (0.342 [95% CI, 0.170-0.532] for cohort A and 0.604 [95% CI, -0.166 to 1.221] for B), and flexible calibration curves in each cohort indicated that the model underestimated true PAS risks on average and that there was evidence of overfitting in both validation cohorts. The use of the model compared to a treat-all strategy by decision curve analysis showed a greater net benefit of the model at a threshold probability of >0.25 in cohort A. However, no net benefit of the model over the treat-all strategy was seen in cohort B at any threshold probability. The performance of the Weiniger model is variable based on the case-mix of the population with regard to PAS clinical risk factors and ultrasound features, highlighting the importance of spectrum bias when applying this PAS prediction model to distinct populations. The model showed benefit for predicting PAS in populations with substantial case-mix heterogeneity at threshold probability of >25%.

OS11.5.A Utility of lumbar puncture (LP) and cerebrospinal fluid (CSF) profile in the diagnosis of neoplastic meningitis (NM): a retrospective cohort study and meta-analyses

Abstract Background CSF cytology remains the gold standard for diagnosing NM, but the sensitivity of this test is low (approximately 50% with one LP). Little is known about the predictive value of routine CSF and clinical findings for a positive cytology. We addressed these questions by analyzing 337 consecutive patients at our institution, and by conducting exhaustive evidence-based literature reviews and meta-analyses of relevant research. Material and Methods 337 consecutive patients at a single institution were evaluated. Of these, 210 met criteria for NM (a positive CSF cytology on at least one of four pre-treatment samples) or no NM (four consecutive negative CSF cytologies). CSF white blood cell count, glucose, protein, and lactate were collected, as well as the presence of neurological symptoms. Optimal threshold values were identified using ROC curves. Sensitivity (Sn), specificity (Sp), and likelihood ratios were calculated for each variable. Two PRISMA-compliant meta-analyses were also performed: one analyzed studies reporting the cytology results of multiple LPs, and one analyzed the frequency of abnormal CSF findings as predictors of NM. Results All published results describe outcomes in patients with known NM. These results are, therefore, strongly distorted by spectrum bias, which artificially inflates the apparent sensitivity and specificity of those outcomes. Thus, 25 studies report the yield of positive CSF cytology on the first, second, and third LP as 73%, 87%, and 94%, but also do not describe the parameters of the procedure. Our series in patients with suspected NM (the cohort most relevant to the practicing physician) reports 74%, 85%, and 92% for immediately processed, 10 mL or greater, lumbar and ventricular specimens. Similarly, while 50 studies suggested that low CSF glucose, high CSF protein, and suggestive symptoms are associated with NM, we found very poor Sn/Sp for elevated protein (61%/61%), low glucose (35%/85%), elevated lactate (35%/91%), and symptoms consistent with NM (74%/73%). Of these values, only an elevated lactate and symptoms were even marginally helpful in predicting a positive CSF cytology. No variable predicted a negative cytology, and 22.8% of patients with a positive CSF cytology had completely normal CSF findings. Conclusion Generally accepted concepts about the frequency of CSF and clinical findings in patients with suspected NM are based on biased estimates in the literature. No CSF chemistry or clinical finding reliably predicts the presence of NM, and almost a quarter of patients with cytologically proven NM have completely normal findings. In patients with positive CSF cytologies, about 73% are detected after one LP, and more than 90% after 3, but again these findings represent overestimates because of pervasive spectrum bias and differences in sample acquisition/handling. Better diagnostic techniques are desperately needed.

CMB lensing reconstruction biases from masking extragalactic sources

Observed cosmic microwave background (CMB) temperature and polarization maps can be powerful cosmological probes and used for CMB lensing reconstruction. However, the CMB maps are inevitably contaminated by foregrounds, some of which are usually masked to perform the analysis. If this mask is correlated to the lensing signal, measurements over the unmasked sky may give biased estimates and hence biased cosmological inferences. For example, masking extragalactic astrophysical emissions associated with objects located in dark matter halos will systematically remove parts of the sky that have a mass density higher than average. This can lead to a modified lensed CMB power spectrum over the unmasked area and biased measurements of lensing reconstruction auto- and cross-correlation power spectra with external matter tracers (both from the direct impact on the lensing power and via modifications to the lensing-dependent reconstruction power spectrum corrections, ${N}_{L}^{(0)}$, ${N}_{L}^{(1)}$, and ${N}_{L}^{(3/2)}$). For direct masking of the CMB lensing field, we give an approximate halo-model prediction of the size of the effect and derive simple analytic models for point sources and threshold masks constructed on a correlated Gaussian foreground field. We show that biases are significantly reduced by optimal filtering of the CMB maps in the lensing reconstruction, which effectively fills back some of the information in small mask holes. We test the resulting lensing power spectrum biases on numerical simulations, masking radio sources, and peaks of thermal Sunyaev-Zeldovich (tSZ) and cosmic infrared background (CIB) emission. For radio point sources, the remaining bias is negligible, but a temperature lensing reconstruction power spectrum bias remains at the 0.5%--5% level if clusters with a mass larger than $\ensuremath{\sim}{10}^{14}\text{ }\text{ }{M}_{\ensuremath{\bigodot}}/h$ are masked or if peaks in the CIB and tSZ maps are removed with ${f}_{\mathrm{sky}}^{\mathrm{mask}}\ensuremath{\simeq}0.5--7%$. In any case, these biases can only be measured with a statistical significance $\ensuremath{\lesssim}2\ensuremath{\sigma}$ for future datasets. Moreover, we quantified the impact of the mask biases in the cross-correlation power spectrum between CMB lensing and tSZ and CIB and found them to be larger (up to $\ensuremath{\sim}30%$). We found that masking tSZ-selected galaxy clusters leads to the largest mask biases, potentially detectable with high significance, and should therefore be avoided as much as possible. For the most realistic masks we considered, masking biases can only be measured with marginal significance. We found that the calibration of cluster masses using CMB lensing, in particular for objects at $z\ensuremath{\lesssim}0.6$, might be significantly affected by mask biases for near-future observations if the lensing signal recovered inside the mask holes is used without further corrections. Conversely, mass calibration of high redshift objects will still deliver unbiased results.

Assessing liver fibrosis in all patients with type 2 diabetes and fatty liver disease: It's time to act now.

Assessing liver fibrosis in all patients with type 2 diabetes and fatty liver disease: It's time to act now.

Vibration and noise suppression method of transformer

Aiming at the problem of noise and vibration of distribution transformer body and base connector under heavy load, the corresponding relationship between vibration and noise, load and power factor of distribution transformer is studied in this paper. In order to improve the operation reliability of distribution transformer, a noise and vibration suppression method based on load scenario optimization is proposed. From the aspects of design reliability, data detection and noise level correction, the typical relationship between the increase of noise level and DC current, noise spectrum and DC bias noise is obtained, and the DC bias limiting measures and correction suggestions are given.

Artificial intelligence (AI) to enhance breast cancer screening: protocol for population-based cohort study of cancer detection

IntroductionArtificial intelligence (AI) algorithms for interpreting mammograms have the potential to improve the effectiveness of population breast cancer screening programmes if they can detect cancers, including interval cancers, without contributing...

Language Screening in 3-Year-Olds: Development and Validation of a Feasible and Effective Instrument for Pediatric Primary Care.

Objective: The study was aimed at evaluating the validity and feasibility of SPES-3 (Sprachentwicklungsscreening), a language screening in 3-year-old children within the constraints of regular preventive medical check-ups.Methods: A four-component screening measure including parental reports on the child's expressive vocabulary and grammar based on the MacArthur Communicative Development Inventory and pediatrician-administered standardized assessments of noun plurals and sentence comprehension was used in a sample of 2,044 consecutively seen children in 30 pediatric offices. One-hundred forty-four children (70 who failed and 74 who passed the screener) comprised the validation sample and also underwent follow-up gold standard assessment. To avoid verification and spectrum bias multiple imputation of missing diagnosis for children who did not undergo gold standard assessment was used. Independent diagnoses by two experts blinded to the screening results were considered gold standard for diagnosing language disorder. Screening accuracy of each of the four subscales was analyzed using receiver operator characteristic (ROC) curves. Feasibility was assessed by use of a questionnaire completed by the pediatricians.Results: The two parental screening subscales demonstrated excellent accuracy with area under the curve (AUC) scores of 0.910 and 0.908 whereas AUC scores were significantly lower for the subscales directly administered by the pediatricians (0.816 and 0.705). A composite score based on both parental screening scales (AUC = 0.946) outperformed single subscales. A cut off of 41.69 on a T-scale resulted in about 20% positive screens and showed good sensitivity (0.878) and specificity (0.876). Practicability, acceptability and sustainability of the screening measure were mostly rated as high.Conclusion: The parent-reported subscales of the SPES-3 language screener are a promising screening tool for use in primary pediatric care settings.

Intense SARS-CoV-2 transmission among affluent Manaus residents preceded the second wave of the epidemic in Brazil

Intense SARS-CoV-2 transmission among affluent Manaus residents preceded the second wave of the epidemic in Brazil

Spectrum Bias and Individual Strengths of SARS-CoV-2 Serological Tests-A Population-Based Evaluation.

Antibody testing for determining the SARS-CoV-2 serostatus was rapidly introduced in early 2020 and since then has been gaining special emphasis regarding correlates of protection. With limited access to representative samples with known SARS-CoV-2 infection status during the initial period of test development and validation, spectrum bias has to be considered when moving from a “test establishment setting” to population-based settings, in which antibody testing is currently implemented. To provide insights into the presence and magnitude of spectrum bias and to estimate performance measures of antibody testing in a population-based environment, we compared SARS-CoV-2 neutralization to a battery of serological tests and latent class analyses (LCA) in a subgroup (n = 856) of the larger population based TiKoCo-19 cohort (n = 4185). Regarding spectrum bias, we could proof notable differences in test sensitivities and specificities when moving to a population-based setting, with larger effects visible in earlier registered tests. While in the population-based setting the two Roche ELECSYS anti-SARS-CoV-2 tests outperformed every other test and even LCA regarding sensitivity and specificity in dichotomous testing, they didn’t provide satisfying quantitative correlation with neutralization capacity. In contrast, our in-house anti SARS-CoV-2-Spike receptor binding domain (RBD) IgG-ELISA (enzyme-linked-immunosorbant assay) though inferior in dichotomous testing, provided satisfactory quantitative correlation and may thus represent a better correlate of protection. In summary, all tests, led by the two Roche tests, provided sufficient accuracy for dichotomous identification of neutralizing sera, with increasing spectrum bias visible in earlier registered tests, while the majority of tests, except the RBD-ELISA, didn’t provide satisfactory quantitative correlations.

Quenching effects in the cumulative jet spectrum

The steeply falling jet spectrum induces a bias on the medium modifications of jet observables in heavy-ion collisions. To explore this effect, we develop a novel analytic framework to study the quenched jet spectrum and its cumulative. We include many energy-loss-related effects, such as soft and hard medium induced emissions, broadening, elastic scattering, jet fragmentation, cone size dependence, and coherence effects. We show that different observables, based on the jet spectrum, are connected, e.g., the nuclear modification, spectrum shift, and the quantile procedure. We present the first predictions for the nuclear modification factor and the quantile procedure with cone size dependence. As a concrete example, we compare dijet and boson+jet events to unfold the spectrum bias effects, and improve quark-, and gluon-jet classification using arguments based on the cumulative. Besides pointing out its flexibility, finally, we apply our framework to other energy loss models such as the hybrid weak/strong-coupling approach.

Spectrum bias in algorithms derived by artificial intelligence: a case study in detecting aortic stenosis using electrocardiograms.

Spectrum bias can arise when a diagnostic test is derived from study populations with different disease spectra than the target population, resulting in poor generalizability. We used a real-world artificial intelligence (AI)-derived algorithm to detect severe aortic stenosis (AS) to experimentally assess the effect of spectrum bias on test performance. All adult patients at the Mayo Clinic between 1 January 1989 and 30 September 2019 with transthoracic echocardiograms within 180 days after electrocardiogram (ECG) were identified. Two models were developed from two distinct patient cohorts: a whole-spectrum cohort comparing severe AS to any non-severe AS and an extreme-spectrum cohort comparing severe AS to no AS at all. Model performance was assessed. Overall, 258 607 patients had valid ECG and echocardiograms pairs. The area under the receiver operator curve was 0.87 and 0.91 for the whole-spectrum and extreme-spectrum models, respectively. Sensitivity and specificity for the whole-spectrum model was 80% and 81%, respectively, while for the extreme-spectrum model it was 84% and 84%, respectively. When applying the AI-ECG derived from the extreme-spectrum cohort to patients in the whole-spectrum cohort, the sensitivity, specificity, and area under the curve dropped to 83%, 73%, and 0.86, respectively. While the algorithm performed robustly in identifying severe AS, this study shows that limiting datasets to clearly positive or negative labels leads to overestimation of test performance when testing an AI algorithm in the setting of classifying severe AS using ECG data. While the effect of the bias may be modest in this example, clinicians should be aware of the existence of such a bias in AI-derived algorithms.

Performance and Limitation of Machine Learning Algorithms for Diabetic Retinopathy Screening: Meta-analysis.

BackgroundDiabetic retinopathy (DR), whose standard diagnosis is performed by human experts, has high prevalence and requires a more efficient screening method. Although machine learning (ML)–based automated DR diagnosis has gained attention due to recent approval of IDx-DR, performance of this tool has not been examined systematically, and the best ML technique for use in a real-world setting has not been discussed.ObjectiveThe aim of this study was to systematically examine the overall diagnostic accuracy of ML in diagnosing DR of different categories based on color fundus photographs and to determine the state-of-the-art ML approach.MethodsPublished studies in PubMed and EMBASE were searched from inception to June 2020. Studies were screened for relevant outcomes, publication types, and data sufficiency, and a total of 60 out of 2128 (2.82%) studies were retrieved after study selection. Extraction of data was performed by 2 authors according to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), and the quality assessment was performed according to the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2). Meta-analysis of diagnostic accuracy was pooled using a bivariate random effects model. The main outcomes included diagnostic accuracy, sensitivity, and specificity of ML in diagnosing DR based on color fundus photographs, as well as the performances of different major types of ML algorithms.ResultsThe primary meta-analysis included 60 color fundus photograph studies (445,175 interpretations). Overall, ML demonstrated high accuracy in diagnosing DR of various categories, with a pooled area under the receiver operating characteristic (AUROC) ranging from 0.97 (95% CI 0.96-0.99) to 0.99 (95% CI 0.98-1.00). The performance of ML in detecting more-than-mild DR was robust (sensitivity 0.95; AUROC 0.97), and by subgroup analyses, we observed that robust performance of ML was not limited to benchmark data sets (sensitivity 0.92; AUROC 0.96) but could be generalized to images collected in clinical practice (sensitivity 0.97; AUROC 0.97). Neural network was the most widely used method, and the subgroup analysis revealed a pooled AUROC of 0.98 (95% CI 0.96-0.99) for studies that used neural networks to diagnose more-than-mild DR.ConclusionsThis meta-analysis demonstrated high diagnostic accuracy of ML algorithms in detecting DR on color fundus photographs, suggesting that state-of-the-art, ML-based DR screening algorithms are likely ready for clinical applications. However, a significant portion of the earlier published studies had methodology flaws, such as the lack of external validation and presence of spectrum bias. The results of these studies should be interpreted with caution.

Diagnostic Accuracy of Point-of-Care Ultrasound for Intussusception: A Multicenter, Noninferiority Study of Paired Diagnostic Tests

To determine the diagnostic accuracy of point-of-care ultrasound (POCUS) performed by experienced clinician sonologists compared to radiology-performed ultrasound (RADUS) for detection of clinically important intussusception, defined as intussusception requiring radiographic or surgical reduction. We conducted a multicenter, noninferiority, observational study among a convenience sample of children aged 3 months to 6 years treated in tertiary care emergency departments across North and Central America, Europe, and Australia. The primary outcome was diagnostic accuracy of POCUS and RADUS with respect to clinically important intussusception. Sample size was determined using a 4-percentage-point noninferiority margin for the absolute difference in accuracy. Secondary outcomes included agreement between POCUS and RADUS for identification of secondary sonographic findings. The analysis included 256 children across 17 sites (35 sonologists). Of the 256 children, 58 (22.7%) had clinically important intussusception. POCUS identified 60 (23.4%) children with clinically important intussusception. The diagnostic accuracy of POCUS was 97.7% (95% confidence interval [CI] 94.9% to 99.0%), compared to 99.3% (95% CI 96.8% to 99.9%) for RADUS. The absolute difference between the accuracy of RADUS and that of POCUS was 1.5 percentage points (95% CI -0.6 to 3.6). Sensitivity for POCUS was 96.6% (95% CI 87.2% to 99.1%), and specificity was 98.0% (95% CI 94.7% to 99.2%). Agreement was high between POCUS and RADUS for identification of trapped free fluid (83.3%, n=40/48) and decreased color Doppler signal (95.7%, n=22/23). Our findings suggest that the diagnostic accuracy of POCUS performed by experienced clinician sonologists may be noninferior to that of RADUS for detection of clinically important intussusception. Given the limitations of convenience sampling and spectrum bias, a larger randomized controlled trial is warranted.

Study Designs: Diagnostic Studies

Diagnostic tests are evolving with betterment of technology, quest for patient safety with less invasive medicine, and evolution of new diseases. It is important to assess diagnostic accuracy of a new test, and clinical impact of introduction of new test on outcomes and cost. A diagnostic study is planned for the index test based on place of new test in diagnostic pathway (screening, triage, diagnostic or add-on test) and established information of the test. A reference standard is used to classify population into diseased and healthy, and the discriminating ability of index test is measured. A sample size is calculated for expected sensitivity/specificity, margin of error and prevalence of disease in population. For dichotomous outcomes, sensitivity, specificity, predictive values and likelihood ratio are used to describe diagnostic accuracy. Efforts should be made to avoid common forms of bias including spectrum bias and partial verification bias, and blinding of observers should preferably be done.