Average Percent Agreement Research Articles

Testing the Ability and Limitations of ChatGPT to Generate Differential Diagnoses from Transcribed Radiologic Findings.

Background The burgeoning interest in ChatGPT as a potentially useful tool in medicine highlights the necessity for systematic evaluation of its capabilities and limitations. Purpose To evaluate the accuracy, reliability, and repeatability of differential diagnoses produced by ChatGPT from transcribed radiologic findings. Materials and Methods Cases selected from a radiology textbook series spanning a variety of imaging modalities, subspecialties, and anatomic pathologies were converted into standardized prompts that were entered into ChatGPT (GPT-3.5 and GPT-4 algorithms; April 3 to June 1, 2023). Responses were analyzed for accuracy via comparison with the final diagnosis and top 3 differential diagnosis provided in the textbook, which served as the ground truth. Reliability, defined based on the frequency of algorithmic hallucination, was assessed through the identification of factually incorrect statements and fabricated references. Comparisons were made between the algorithms using the McNemar test and a generalized estimating equation model framework. Test-retest repeatability was measured by obtaining 10 independent responses from both algorithms for 10 cases in each subspecialty, and calculating the average pairwise percent agreement and Krippendorff α. Results A total of 339 cases were collected across multiple radiologic subspecialties. The overall accuracy of GPT-3.5 and GPT-4 for final diagnosis was 53.7% (182 of 339) and 66.1% (224 of 339; P < .001), respectively. The mean differential score (ie, proportion of top 3 diagnoses that matched the original literature differential diagnosis) for GPT-3.5 and GPT-4 was 0.50 and 0.54 (P = .06), respectively. Of the references provided in GPT-3.5 and GPT-4 responses, 39.9% (401 of 1006) and 14.3% (161 of 1124; P < .001), respectively, were fabricated. GPT-3.5 and GPT-4 generated false statements in 16.2% (55 of 339) and 4.7% (16 of 339; P < .001) of cases, respectively. The range of average pairwise percent agreement across subspecialties for the final diagnosis and top 3 differential diagnosis was 59%-98% and 23%-49%, respectively. Conclusion ChatGPT achieved the best results when the most up-to-date model (GPT-4) was used and when it was prompted for a single diagnosis. Hallucination frequency was lower with GPT-4 than with GPT-3.5, but repeatability was an issue for both models. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Chang in this issue.

Analytical validation of the Labcorp Plasma Complete test to enable precision oncology through solid tumor liquid biopsy comprehensive genomic profiling.

3063 Background: To help guide treatment decisions and enable clinical trial matching for cancer patients, tumor genomic profiling is an essential precision oncology tool. Liquid biopsy may be used as a complementary approach to assess tumor-specific DNA alterations circulating in the blood. The Labcorp Plasma Complete test is a next-generation-sequencing (NGS), cell-free DNA (cfDNA) comprehensive genomic profiling test that identifies actionable and clinically relevant variants in advanced and metastatic solid cancers across 521 genes. Methods: The Labcorp Plasma Complete test is a hybrid capture based, targeted NGS test, optimized for 25 ng of cfDNA input and is intended for variant detection in all solid cancer indications. The test interrogates all coding exons of 521 genes to comprehensively detect single nucleotide variants (SNVs) and insertion and deletions (indels). The test also reports copy number amplifications (CNAs) and gene translocations in 12 genes, and microsatellite instability (MSI). In total, 631 cancer patient plasma samples were evaluated by the test including lung, breast, colorectal, head and neck, pancreatic, gynecologic, and prostate cancers among other indications. Well-characterized reference samples (Genome in a Bottle) were utilized to assess analytical specificity. Contrived cell lines and clinical samples were utilized to establish analytical sensitivity, accuracy, as well as precision, reproducibility, and repeatability (PRR). Results: The test yielded results for 98.3% (620/631) of cases and identified variants in 94.7% (587/620) with a median of 6 variants per case. Clinically actionable variants were detected in 42.1% of clinical samples. Analytical specificity was ≥99.9999% for SNVs and 100% for indels, CNAs, translocations, and MSI. Analytical sensitivity (limit of detection, 95%) was verified for each variant type, with a median variant allele frequency (VAF) of 1.42% for SNVs and 1.43% for indels, 1.7-fold for amplifications, 0.35% fusion read fraction for translocations, and 0.82% VAF for MSI. The intra-laboratory PRR study resulted in 94.9% average percent agreement (APA) and 99.9% average negative agreement (ANA) for sequence variants (SNVs and indels) and 100% APA and ANA for CNAs, translocations, and MSI. Orthogonal assays including other NGS tests and digital PCR were utilized to assess Labcorp Plasma Complete accuracy which demonstrated an aggregate analytical concordance of 96.28% positive percent agreement and &gt;99.9999% negative percent agreement for all variants. Conclusions: The analytical validation of the Labcorp Plasma Complete test demonstrates that the liquid biopsy approach is highly sensitive, specific, accurate, reproducible, and robust for comprehensive genomic profiling to complement tissue-based testing and inform clinical decision making.

Validation of a shortened MR imaging protocol for pediatric spinal pathology.

Conventional pediatric spine MRI protocols have multiple sequences resulting in long acquisition times. Sedation is consequently required. This study evaluates the diagnostic capability of a limited MRI spine protocol for selected common pediatric indications. Spine MRIs at CHEO between 2017 and 2020 were reviewed across pediatric patients younger than four years old. Two blinded neuroradiologists reviewed limited scan sequences, and results were independently compared to previously reported findings from the complete imaging series. T2 sagittal sequences from the craniocervical junction to sacrum and T1 axial sequence of the lumbar spine constitute the short protocol, with the outcomes of interest being cerebellar ectopia, syrinx, level of conus, filum < 2mm, fatty filum, and spinal dysraphism. A total of 105 studies were evaluated in 54 male and 51 female patients (mean age 19.2months). The average combined scan time of the limited sequences was 15min compared to 35min for conventional protocols (delta = 20min). The average percent agreement between full and limited sequences was > 95% in all but identifying a filum < 2mm, where the percent agreement was 87%. Using limited MR sequences had high sensitivity (> 0.91) and specificity (> 0.99) for the detection of cerebellar ectopia, syrinx, fatty filum, and spinal dysraphism. This study demonstrates that selected spinal imaging sequences allow for consistent and accurate diagnosis of specific clinical conditions. A limited spine imaging protocol has potential as a screening test to reduce the need for full-sequence MRI scans. Further work is needed to determine utility of selected imaging for other clinical indications.

Age and growth assessment of the near-threatened fish Rita rita (Siluriformes: Bagridae) in the Ganges basin

Introduction: Rita rita is a freshwater catfish under threat of extinction, mainly from loss of breeding and nursing grounds. A reliable method for age and growth estimation is needed by fishery managers. Objective: To identify the best body structure for age and growth estimation. Methods: We assessed estimates precision based on Average Percent Error (APE), Coefficient Variation (CV), and Percent Agreement (PA) between readers separately analyzed each calcified structure. We used 390 fish samples from three rivers, Ganga, Yamuna and Ramganga, from September 2018 to August 2019. Results: The three indicators favored the use of vertebrae for age estimation; the growth band seems to be annual and formed from May to September. The growth equations were Lt = 90.19(1-e-0.145(t+0.51)) for Ganga; Lt = 91.19 (1-e-0.14(t+0.59)) for Yamuna and Lt = 89.63 (1-e-0.15(t+0.68)) for Ramganga. Conclusion: This species reaches moderate growth in these rivers, where vertebrae are the recommended age estimation structure, followed, in case of need, by sectioned otoliths, whole otoliths and opercular bones. Pectoral spines should be avoided, especially in older fish.

Interrater Reliability of Expert Electroencephalographers Identifying Seizures and Rhythmic and Periodic Patterns in EEGs.

The validity of brain monitoring using electroencephalography (EEG), particularly to guide care in patients with acute or critical illness, requires that experts can reliably identify seizures and other potentially harmful rhythmic and periodic brain activity, collectively referred to as "ictal-interictal-injury continuum" (IIIC). Previous interrater reliability (IRR) studies are limited by small samples and selection bias. This study was conducted to assess the reliability of experts in identifying IIIC. This prospective analysis included 30 experts with subspecialty clinical neurophysiology training from 18 institutions. Experts independently scored varying numbers of ten-second EEG segments as "seizure (SZ)," "lateralized periodic discharges (LPDs)," "generalized periodic discharges (GPDs)," "lateralized rhythmic delta activity (LRDA)," "generalized rhythmic delta activity (GRDA)," or "other." EEGs were performed for clinical indications at Massachusetts General Hospital between 2006 and 2020. Primary outcome measures were pairwise IRR (average percent agreement [PA] between pairs of experts) and majority IRR (average PA with group consensus) for each class and beyond chance agreement (κ). Secondary outcomes were calibration of expert scoring to group consensus, and latent trait analysis to investigate contributions of bias and noise to scoring variability. Among 2,711 EEGs, 49% were from women, and the median (IQR) age was 55 (41) years. In total, experts scored 50,697 EEG segments; the median [range] number scored by each expert was 6,287.5 [1,002, 45,267]. Overall pairwise IRR was moderate (PA 52%, κ 42%), and majority IRR was substantial (PA 65%, κ 61%). Noise-bias analysis demonstrated that a single underlying receiver operating curve can account for most variation in experts' false-positive vs true-positive characteristics (median [range] of variance explained ([Formula: see text]): 95 [93, 98]%) and for most variation in experts' precision vs sensitivity characteristics ([Formula: see text]: 75 [59, 89]%). Thus, variation between experts is mostly attributable not to differences in expertise but rather to variation in decision thresholds. Our results provide precise estimates of expert reliability from a large and diverse sample and a parsimonious theory to explain the origin of disagreements between experts. The results also establish a standard for how well an automated IIIC classifier must perform to match experts. This study provides Class II evidence that an independent expert review reliably identifies ictal-interictal injury continuum patterns on EEG compared with expert consensus.

Building Consensus for a Shared Definition of Adverse Events: A Case Study in the Profession of Dentistry.

To achieve high-quality health care, adverse events (AEs) must be proactively recognized and mitigated. However, there is often ambiguity in applying guidelines and definitions. We describe the iterative calibration process needed to achieve a shared definition of AEs in dentistry. Our alignment process includes both independent and consensus building approaches. We explore the process of defining dental AEs and the steps necessary to achieve alignment across different care providers. Teams from 4 dental institutions across the United States iteratively reviewed patient records after identification of charts using an automated trigger tool. Calibration across teams was supported through negotiated definition of AEs and standardization of evidence provided in review. Interrater reliability was assessed using descriptive and κ statistics. After 5 iterative cycles of calibration, the teams (n = 8 raters) identified 118 cases. The average percent agreement for AE determination was 82.2%. Furthermore, the average, pairwise prevalence and bias-adjusted κ (PABAK) was 57.5% (κ = 0.575) for determining AE presence. The average percent agreement for categorization of the AE type was 78.5%, whereas the PABAK was 48.8%. Lastly, the average percent agreement for categorization of AE severity was 82.2% and the corresponding PABAK was 71.7%. Successful calibration across reviewers is possible after consensus building procedures. Higher levels of agreement were found when categorizing severity (of identified events) rather than the events themselves. Our results demonstrate the need for collaborative procedures as well as training for the identification and severity rating of AEs.

Implementation fidelity, attitudes, and influence: a novel approach to classifying implementer behavior

BackgroundThe current study sought to (1) describe a new classification approach for types of implementer behavior and (2) explore the implementer behavior change in response to tailored implementation facilitation based on the classifications.MethodsA small-scale, cluster-randomized hybrid type III implementation trial was conducted in 38 early care and education classrooms that were part of the Together, We Inspire Smart Eating (WISE) program. WISE focuses on 4 evidence-based practices (EBPs), which are implemented by teachers to promote nutrition. External facilitators (N = 3) used a modified Rapid Assessment Procedure Informed Clinical Ethnography (RAPICE) to complete immersion (i.e., observations) and thematic content analyses of interviews to identify the characteristics of teachers’ behavior at varying levels of implementation fidelity. Three key factors—attitudes toward the innovation, fidelity/adaptations, and influence—were identified that the research team used to classify teachers’ implementation behavior. This process resulted in a novel classification approach. To assess the reliability of applying the classification approach, we assessed the percent agreement between the facilitators. Based on the teachers’ classification, the research team developed a tailored facilitation response. To explore behavior change related to the tailored facilitation, change in fidelity and classification across the school year were evaluated.ResultsThe classifications include (1) enthusiastic adopters (positive attitude, meeting fidelity targets, active influence), (2) over-adapting adopters (positive attitude, not meeting fidelity targets, active influence), (3) passive non-adopters (negative attitude, not meeting fidelity targets, passive influence), and (4) active non-adopters (negative attitudes, not meeting fidelity targets, active influence). The average percent agreement among the three facilitators for classification was 75%. Qualitative data support distinct patterns of perceptions across the classifications. A positive shift in classification was observed for 67% of cases between the mid-point and final classification. Finally, we generated an expanded classification approach to consider additional combinations of the three factors beyond those observed in this study.ConclusionsData from this study support the ability to apply the classification approach with moderate to high reliability and to use the approach to tailor facilitation toward improved implementation. Findings suggest the potential of our approach for wider application and potential to improve tailoring of implementation strategies such as facilitation.

Validation of a multi-modal blood-based test for the detection of colorectal cancer with sub single molecule sensitivity.

3627 Background: Blood-based colorectal cancer (CRC) screening tests can improve adherence to screening guidelines. Yet, current commercially available options have poor sensitivity and specificity inhibiting incorporation into routine clinical care. Here we report the validation of a blood-based test for the detection of colorectal cancer and advanced neoplasia. Methods: This blood-based test aims to detect colorectal neoplasia by identifying tumor-associated biomarkers including genomic or epigenomic (methylation and fragmentomics) signatures in cell-free DNA (cfDNA). cfDNA is partitioned based on methylation level, enriched for informative genomic regions, and sequenced. This novel workflow enables high-fidelity analysis of multi-modal information in majority of extracted cfDNA molecules. Results are integrated into a binary “detected” versus “not-detected” result using a proprietary bioinformatic pipeline (Guardant Health, USA). The assay was trained on samples obtained from &gt;6,000 unique individuals (2,685 cancer-free and 1,698 with advanced colorectal neoplasia (ACN) for training, 1,072 cancer-free and 551 with ACN for threshold setting). The thresholds were frozen prior to validation targeting a specificity of &gt; 91.5%. Each aspect of the validation study followed Nex-StoCT CLIA working group and CLSI guidelines. Results: Limit of detection (LoD) was established across six dilutions. Even for low cfDNA mass inputs of less than 4ng, the 95% LoD was determined to be less than 1 tumor-derived genomic equivalent (0.5), indicating over at least 10-fold increase in assay sensitivity compared to best-in-class assays for somatic mutation detection. Precision studies in 60 positive and negative replicates from clinical samples yielded &gt;90% average positive and negative percent agreement both within and between batches. Endogenous interference studies yielded &gt; 90% positive and negative percent agreement between reference control and common endogenous substances, including albumin, bilirubin, hemoglobin, triglycerides, and genomic DNA, in clinical positive and negative samples and minimally manipulated samples. The clinical validation of the test was conducted in &gt; 300 cases (biobanked pre-operative cohort for CRC cases and screening cohort for advanced adenoma and negative cases). Conclusions: Here we present the validation of a multi-modal blood-based test for the detection of colorectal cancer. This test is currently being evaluated in a registrational study (ECLIPSE: NCT04136002).

Validity and reliability of a questionnaire measuring EBDPs among registered dietitian nutritionist.

There is a lack of a valid and reliable instrument that measures objective and subjective knowledge of evidence-based dietetic practices (EBDP) among registered dietitian nutritionists (RDNs). The present study aimed to develop and assess the validity and reliability of an EBDP Questionnaire (EBDPQ) with objective knowledge items (i.e., quiz items) among RDNs in the USA. Subscales from four existing evidence-based practice (EBP) instruments were combined and modified for dietetics. Content and face validity and internal consistency were used to assess the full survey. Survey responsiveness and test-retest reliability were evaluated within the objective knowledge subscale. Content experts (n = 11) and nutrition professionals (n = 16) completed validation reviews. Doctoral students enrolled in a research course (n = 12) were used to analyse survey responsiveness. Internal and test-retest relability analyses utilised RDN participants (time point 1, n = 482; time point 2, n = 335). Content validation resulted in a 38-item questionnaire. Average percent agreement among face validity reviewers was 95.1%. Only the pre-validation version of the objective knowledge subscale resulted in significantly higher post-course scores (Mdn = 11.50) compared to the pre-course (Mdn = 9.75, p = 0.05). The validated instrument had excellent internal consistency (Cronbach's α = 0.91); however, the objective knowledge subscale was low (Cronbach's α = 0.41). A good degree of reliability was found between the two time points (intraclass correlation coefficient = 0.71). The EBDPQ demonstrated adequate validity and reliability among RDNs. Future research should assess construct validity, with the responsiveness and objective knowledge subscale requiring additional evaluation through an EBDP course.

Predicting the objective and priority of issue reports in software repositories

Software repositories such as GitHub host a large number of software entities. Developers collaboratively discuss, implement, use, and share these entities. Proper documentation plays an important role in successful software management and maintenance. Users exploit Issue Tracking Systems, a facility of software repositories, to keep track of issue reports, to manage the workload and processes, and finally, to document the highlight of their team’s effort. An issue report is a rich source of collaboratively-curated software knowledge, and can contain a reported problem, a request for new features, or merely a question about the software product. As the number of these issues increases, it becomes harder to manage them manually. GitHub provides labels for tagging issues, as a means of issue management. However, about half of the issues in GitHub’s top 1000 repositories do not have any labels. In this work, we aim at automating the process of managing issue reports for software teams. We propose a two-stage approach to predict both the objective behind opening an issue and its priority level using feature engineering methods and state-of-the-art text classifiers. To the best of our knowledge, we are the first to fine-tune a Transformer for issue classification. We train and evaluate our models in both project-based and cross-project settings. The latter approach provides a generic prediction model applicable for any unseen software project or projects with little historical data. Our proposed approach can successfully predict the objective and priority level of issue reports with $$82\%$$ (fine-tuned RoBERTa) and $$75\%$$ (Random Forest) accuracy, respectively. Moreover, we conducted human labeling and evaluation on unlabeled issues from six unseen GitHub projects to assess the performance of the cross-project model on new data. The model achieves $$90\%$$ accuracy on the sample set. We measure inter-rater reliability and obtain an average Percent Agreement of $$85.3\%$$ and Randolph’s free-marginal Kappa of 0.71 that translate to a substantial agreement among labelers.

Validity of the Novel Radiological Classification System of the Distal Femur.

Aseptic loosening (AL) is among the most important causes of failure after total knee arthroplasty (TKA). However, while there are numerous underlying causes of AL, the morphometry of the distal femur and intramedullary canal has not been sufficiently demonstrated. This study aimed to show the interobserver and intraobserver reliability and validity of the Citak classification, which has been recently defined according to the morphometry of the distal femur and provides a risk factor definition for AL. A total of 200 patients whose standardized anteroposterior (AP) and lateral images of the knee joint were obtained between October 2019 and April 2020 were retrospectively evaluated in this study. Patients with a history of extra-articular deformity and knee surgery were excluded from the study. For AL, morphologies of the distal femur were identified by two observers using the new radiological classification system of the distal femur. Mean pairwise Cronbach's alpha coefficient was used to assess the intra- and interobserver agreement of the classification. There was excellent interobserver agreement for the 20cm proximal and 2cm proximal to the lateral joint line (PLJL) and adductor tubercle (PAD), respectively. The mean Cronbach's alpha coefficient was 0.96 (range 0.764-0.944) for the PAD and 0.98 (range 0.734-0.929) for the PLJL. There was also an excellent intraobserver agreement, with 93% average pairwise percent agreement for the index group and 95.5% average pairwise percent agreement for the anatomical classification group. The level of inter- and intraobserver agreement for the morphology of the distal femur was excellent in the new radiological classification system, which was shown to be beneficial in the planning of revision knee arthroplasty for AL. However, there is a need for further studies in order to make a correlation of the classification with specific intraoperative findings.

Immediate Physiological Responses to Inspiratory Flow Limited Events in Mild Obstructive Sleep Apnea.

Rationale: Inspiratory flow limitation (IFL), characterized by flattening of individual breaths on the airflow/time tracing, is a noninvasive indicator of elevated upper airway resistance. An IFL "event" in isolation has not been defined, nor has the ability to reproducibly identify event occurrence been tested. IFL events and their association with immediate physiological responses-as well as the impact of characteristics such as age, sex, sleep stage, sleepiness, and event duration on their association with such outcomes-have not been studied. Symptomatic patients with a normal to mildly abnormal apnea-hypopnea index who have predominant IFL on their polysomnography may benefit from treatment. Objectives: To test the reproducibility of identifying IFL events and their termination and to determine the frequency of the immediate physiological response to their occurrence, including desaturation, electroencephalography (EEG) arousal, and increased heart rate (HR). Methods: Fifty-eight patients with obstructive sleep apnea (OSA) underwent full diagnostic polysomnography. IFL events and their termination were identified manually using predefined rules from the unscored nasal cannula flow channel alone and were evaluated for responses such as EEG arousal, oxygen desaturation of ⩾3%, and HR increase. Results: Interscorer reliability was acceptable, with an average percent agreement for occurrence of 82% ± 3%. Of all IFL events, 24% (regardless of the definition) were not associated with an EEG arousal, an increase in HR, or O2 desaturation. Of all IFL events scored, 25% caused O2 desaturation, 40% were associated with an EEG arousal, and 55% were associated with an increase in HR; 67% caused either an EEG arousal and/or an increase in HR. Responses were observed to occur either in isolation or in combination. IFL events that terminated with at least two non-IFL breaths, one of which had a 200% increase in amplitude, were significantly associated with O2 desaturation, EEG arousal, and increase in HR compared with events that ended in one non-IFL breath. IFL events that had a >50% reduction in flow amplitude compared with baseline were significantly associated with O2 desaturation compared with events that had a 30% reduction or less. Conclusions: Most IFL events resulted in immediate physiological responses, and no single consequence reliably occurred after every event. We propose a framework that can incorporate the scoring of IFL events into assessing the diagnosis and severity of OSA and suggest that no single consequence be used to define IFL as a respiratory event. The relationship of IFL events to OSA outcomes remains to be tested.

Developing and Evaluating a Remote Quality Assurance System for Point-of-Care Ultrasound for an Internal Medicine Residency Global Health Track.

Background: A quality assurance system is vital when using point-of-care ultrasound (POCUS) to ensure safe and effective ultrasound use. There are many barriers to implementing a quality assurance system including need for costly software, faculty time, and extra work to log images. Methods: With minimal funding or protected faculty time, we successfully developed an effective remote quality assurance system between residents rotating internationally and faculty in the US. Results: 270 total exams were logged using this system (41 per resident over a 7 week period). Over the course of the implementation period, a significant increase was seen in average image quality (p = 0.030) and percent agreement with reviewer (p = 0.021). No significant increase was seen for percent images with quality rating 5/5 (p = 0.068) or for studies per resident per week (p = 0.30). Discussion/Conclusions: A quality assurance system for remote review and feedback of POCUS exams was successfully developed with limited available funding, using consumer-level software and an educational collaboration. Residents used the system regularly and demonstrated improvement in reviewer-rated image acquisition and interpretation skills. A similar system can be applied for physicians in any geographic area looking to learn POCUS, in partnership with local or international POCUS mentors. We detail a step-by-step approach, challenges encountered, and lessons learned, to help guide others seeking to implement similar programs.

Using Bumble Bee Watch to investigate the accuracy and perception of bumble bee (Bombus spp.) identification by community scientists.

Community science programs provide an opportunity to gather scientific data to inform conservation policy and management. This study examines the accuracy of community science identifications submitted to the North American Bumble Bee Watch program on a per species level and as compared to each species’ conservation status, as well as users (members of the public) and experts (those with expertise in the field of bumble bee biology) perceived ease of species identification. Photos of bumble bees (Hymenoptera: Apidae: Bombus) are submitted to the program by users and verified (species name corrected or assigned as necessary) by an expert. Over 22,000 records from over 4,900 users were used in the analyses. Accuracy was measured in two ways: percent agreement (percent of all records submitted correctly by users) and veracity (percent of all verified records submitted correctly by the users). Users generally perceived it harder to identify species than experts. User perceptions were not significantly different from the observed percent agreement or veracity, while expert perceptions were significantly different (overly optimistic) from the observed percent agreement but not the veracity. We compared user submitted names to final expert verified names and found that, for all species combined, the average percent agreement was 53.20% while the average veracity was 55.86%. There was a wide range in percent agreement values per species, although sample size and the role of chance did affect some species agreements. As the conservation status of species increased to higher levels of extinction risk, species were increasingly more likely to have a lower percent agreement but higher levels of veracity than species of least concern. For each species name submitted, the number of different species verified by experts varied from 1 to 32. Future research may investigate which factors relate to success in user identification through community science. These findings could play a role in informing the design of community science programs in the future, including for use in long-term and national-level monitoring of wild pollinators.

Interobserver Reliability of the Coronary Artery Disease Reporting and Data System in Clinical Practice.

This study aimed to evaluate interobserver reproducibility between cardiothoracic radiologists applying the Coronary Artery Disease Reporting and Data System (CAD-RADS) to describe atherosclerotic burden on coronary computed tomography angiography. Forty clinical computed tomography angiography cases were retrospectively and independently evaluated by 3 attending and 2 fellowship-trained cardiothoracic radiologists using the CAD-RADS lexicon. Radiologists were blinded to patient history and underwent initial training using a practice set of 10 subjects. Interobserver reproducibility was assessed using an intraclass correlation (ICC) on the basis of single-observer scores, absolute agreement, and a 2-way random-effects model. Nondiagnostic studies were excluded. ICC was also performed for CAD-RADS scores grouped by management recommendations for absent (0), nonobstructive (1 to 2), and potentially obstructive (3 to 5) CAD. Interobserver reproducibility was moderate to good (ICC: 0.748, 95% confidence interval [CI]: 0.639-0.842, P<0.0001), with higher agreement among cardiothoracic radiology fellows (ICC: 0.853, 95% CI: 0.730-0.922, P<0.0001) than attending radiologists (ICC: 0.711, 95% CI: 0.568-0.824, P<0.0001). Interobserver reproducibility for clinical management categories was marginally decreased (ICC: 0.692, 95% CI: 0.570-0.802, P<0.0001). The average percent agreement between pairs of radiologists was 84.74%. Percent observer agreement was significantly reduced in the presence (M=62.22%, SD=15.17%) versus the absence (M=80.91%, SD=17.97%) of modifiers, t(37.95)=3.566, P=0.001. Interobserver reliability and agreement with the CAD-RADS terminology are moderate to good in clinical practice. However, further investigations are needed to characterize the causes of interobserver disagreement that may lead to differences in management recommendations.

A Pilot Study on Fetal Heart Rate Extraction from Wearable Abdominal Inertial Sensors

This paper proposes a novel approach for detecting fetal heart rate (FHR) using seismo-cardiogram (SCG) and gyro-cardiogram (GCG) recordings collected from abdominal inertial sensors. A proof-of-concept setup with commercially available sensor nodes is prepared. The FHR components are extracted from the fused cepstrum of recordings of all the sensors. The feasibility of the proposed method is evaluated with experiments on ten pregnant women under supine, seated, and standing positions. The results are compared with simultaneously-collected recordings of fetal cardiotocography (fCTG). The best position for collecting the signals is deemed to be the supine position, which reports best average root mean square error (RMSE) of 9.83 BPM and average positive percent agreement (PPA) of 84.44% for the SCG signal. The overall results of RMSE are 11.40 BPM from SCG and 12.08 BPM from GCG. The overall reliability from SCG is 75.02%, which is slightly lower than the value of 75.52% from GCG. In summary, the results are comparable between the two modalities, suggesting no significant difference between the usage of the two methods. Our results indicate that wearable inertial sensors could potentially be used to extract FHR outside the clinic with accuracy and reliability metrics comparable to other modalities such as fCTG.

Reproducibility of Grading in Chronic Intervillositis of Unknown Etiology.

Chronic intervillositis of unknown etiology (CIUE) is a rare placental inflammatory process associated with pregnancy loss and recurrence. We conducted a quality assurance study to assess the diagnostic accuracy and reproducibility of CIUE grading at our institution. Hematoxylin and eosin-stained slides from 20 CIUE cases (31 slides) were reviewed by 7 perinatal pathologists in 2 sequential rounds. Reviewers were instructed to use the diagnostic criteria they were presently following for CIUE and to grade each slide according to the Rota scheme. In the first round, 20 slides were assessed. The diagnostic accuracy was 94%, the average percent agreement of Rota grade was 79%, and the Fleiss' kappa value for interobserver variability was 0.54. The results were reviewed by all pathologists with diagnostic and grading criteria agreed upon prior to the second round. In round 2, the remaining 11 slides were assessed. Diagnostic accuracy was 83%, the average percent agreement on Rota grade was 70%, and the Fleiss' kappa value for interobserver variability was 0.36. Overall, diagnostic accuracy was high and agreement on Rota grade was moderate. Group review did not appear to improve accuracy. Simplifying CIUE grading to a low-grade/high-grade scheme (<50% or ≥50%) might improve grading reproducibility.

140 Use of GoPro to Assess Central Venous Line Placement Competency Asynchronously

Complications due to failed central venous access (CVA) attempts are associated with increased morbidity and health care costs. The Agency for Healthcare Research and Quality (AHRQ) estimates a central line associated blood stream infection (CLABSI) costs approximately $70,000. There is a high failure rate in completing ultrasound guided central venous access (UGCVA) procedures among first year resident interns. Training and assessment is important, however physician time to assess resident competency is limited. Previous research has demonstrated that a GoPro camera can be used to capture procedures. In this research, we will evaluate the use of video recordings to asynchronously evaluate resident procedural competency in ultrasound guided central venous line (USGCVL) placement. This research had several steps. First, we evaluated the ability of GoPro cameras to capture procedural competencies during USGCVL procedures. Next, we developed and validated an evaluation rubric consisting of critical competencies identified from the literature and institutional guidelines. Faculty reviewers evaluated the GoPro videos asynchronously for visual clarity, and completeness of observation while using the rubric. Results demonstrated high agreement between reviewers (80-100%) for visualization of each competency. Residents were recruited to participate in this study during a training course designed to assess procedural competency in central line placement using a cadaver. Following consent, participants donned a head-mounted GoPro camera to record their performance. During the procedure, faculty evaluated residents' performance synchronously (via Qualtrics) using the same evaluation rubric validated in the prior study. At a later date, two teams each composed of a surgeon and emergency medicine (EM) physician (different than the synchronous reviewers) asynchronously viewed and independently evaluated videos divided into groups of four using the same rubric via Qualtrics. Inter-rater reliability was calculated between pairs of asynchronous reviewers and average percent agreement was calculated for both the asynchronous and the asynchronous versus live reviews. Video recordings of twenty-eight USGCVL procedures were collected. Twenty-four met study inclusion criteria (1080p 30fps continuous video no significant glare, no gross misalignment of the video field of view) and twenty are included in the data analysis. (Four videos have not been reviewed by their assigned reviewers yet.) Results are presented in Table 1. Overall percent agreement approached 80% for both asynchronous and asynchronous versus live comparisons. This study supports the feasibility of using a head mounted GoPro camera to record video for asynchronous evaluation of resident USGCVL placement. Continued refinement of data capture, scoring practices, and the grading rubric is needed to increase both the validity and reliability of the assessment process.Table 1Average percent agreement.Average Percent Agreement (%)95% CI (%)Asynchronous Rater Comparison 12 VideosRater 2 (surgeon) vs. Rater 3 (EM)73.4744.04100Live Graders vs. Asynchronous 8 VideosRater 1, 3 (EM) vs. live graders78.1242.54100 Open table in a new tab

Diagnostic validity of the MINI-KID disorder classifications in specialized child and adolescent psychiatric outpatient clinics in Sweden

BackgroundMissing diagnostic information often results poor accuracy of the clinical diagnostic decision process. The Mini International Neuropsychiatric Interview for Children and Adolescents (MINI-KID) is a short standardized diagnostic interview and covers a rather broad range of diagnoses applicable to children and adolescents. MINI-KID disorder classifications have shown test-retest reliability and validity comparable to other standardized diagnostic interviews and is claimed to be a useful tool for diagnostic screening in Child and Adolescent Psychiatric care. The concordance between the Swedish language version of the MINI-KID Interview and LEAD (Longitudinal, Expert, All Data) research diagnoses was studied in secondary child and adolescent psychiatric outpatient care.MethodsMINI-KID interviews were performed for 101 patients, boys n = 50, girls n = 51, aged 4 to 18 years. The duration of the interview was on average 46 min, the child/adolescent participating together with the parent(s) in most cases. The seven most prevalent diagnoses were included in the analyses.ResultsThe average overall percent agreement (OPA) between MINI-KID and LEAD was 79.5%, the average percent positive agreement (PPA) 35.4 and the average percent negative agreement (NPA) 92.7. OPA was highest for Obsessive-Compulsive Disorder (OCD) (0.89), Tic disorders (0.88) and Pervasive developmental disorders (0.81). There were similar results in diagnostic agreement comparing the two versions: the standard MINI-KID and MINI-KID for parents. The specific screening questions in MINI-KID resulted in additional preliminary diagnoses compared with the regular initial clinical assessment.ConclusionsOverall, there was an acceptable agreement between MINI-KID disorder classifications and research diagnoses according to LEAD. The standardized interview MINI-KID could be considered as a tool with the possibility to give valuable information in the diagnostic process in child and adolescent care which is similar to the setting in the present study.

Resident training and interrater agreements using the ACNS critical care EEG terminology

PurposeElectroencephalography (EEG) remains the gold standard for identifying rhythmic and periodic patterns in critically ill patients. Residents have frequent exposures to EEG and critically ill patients during their training. Our study aimed to assess resident competency in the use of the American Clinical Neurophysiology Society (ACNS) critical care EEG terminology. MethodsAfter self-guided reading and a 2-hour session reviewing the ACNS critical care EEG Terminology training slides, 16 adult neurology residents (PGY 2–4) completed the ACNS certification test. Performance scores were reported as average percent agreement (PA%) with a previously established 5-member expert panel. Interrater agreement was calculated to gauge consensus among peers within the resident cohort. Self-reported comfort levels using the terminology were also obtained. ResultsThe overall pass rate for our cohort was 50% and the median score was 74%. The terms with the highest PA% were: seizures (86.4%), main term 1 (78%), main term 2 (74%). Interrater agreement scores (kappa values) were almost perfect for seizure, and substantial for main terms 1 and 2. ConclusionsOur data suggests that with minimal investment, adult neurology residents at various stages of training can effectively learn the ACNS critical care EEG Terminology.