Inter-rater Agreement Analysis Research Articles

Accuracy of the LaserSAFE technique for detecting positive surgical margins during robot-assisted radical prostatectomy: blind assessment and inter-rater agreement analysis.

Fluorescence confocal microscopy (FCM) is a new imaging modality capable of generating digital microscopic resolution scans of fresh surgical specimens, and holds potential as an alternative to frozen section (FS) analysis for intra-operative assessment of surgical margins. Previously, we described the LaserSAFE technique as an application of FCM for margin assessment in robot-assisted radical prostatectomy (RARP) using the Histolog® scanner. This study describes the accuracy and inter-rater agreement of FCM imaging compared to corresponding paraffin-embedded analysis (PA) among four blinded pathologists for the presence of positive surgical margins (PSM). RARP specimens from patients enrolled in the control arm of the NeuroSAFE PROOF study (NCT03317990) were analysed from April 2022 to February 2023. Prostate specimens were imaged using the Histolog® scanner before formalin fixation and PA. Four trained assessors, blinded to PA, reviewed and analysed FCM images of the posterolateral prostatic surface. A total of 31 prostate specimens were included in the study. PA per lateral side of the prostate identified 11 instances of positive margins. Among the four histopathologists included in our study, FCM achieved a sensitivity of 73-91 and specificity of 94-100% for the presence of PSM. Fleiss' Kappa for inter-rater agreement on PSM was 0.78 (95% confidence interval = 0.64-0.92), indicating substantial agreement. This blinded analysis of FCM versus PA among histopathologists with different experience levels demonstrated high accuracy and substantial inter-rater agreement for diagnosing PSM. This supports the role of the FCM as an alternative to FS.

Artificial intelligence's suggestions for level of amputation in diabetic foot ulcers are highly correlated with those of clinicians, only with exception of hindfoot amputations.

Diabetic foot ulcers (DFUs) are a growing public health problem, paralleling the increasing incidence of diabetes. While prevention is most effective treatment for DFUs, challenge remains on selecting the optimal treatment in cases with DFUs. Health sciences have greatly benefited from the integration of artificial intelligence (AI) applications across various fields. Regarding amputations in DFUs, both literature and clinical practice have mainly focused on strategies to prevent amputation and identify avoidable risk factor. However, there are very limited data on assistive parameters/tools that can be used to determine the level of amputation. This study investigated how well ChatGPT, with its lately released version 4o, matches the amputation level selection of an experienced team in this field. For this purpose, clinical photographs from patients who underwent amputations due to diabetic foot ulcers between May 2023 and May 2024 were submitted to the ChatGPT-4o program. The AI was tasked with recommending an appropriate amputation level based on these clinical photographs. Data from a total of 60 patients were analysed, with a median age of 64.5 years (range: 41-91). According to the Wagner Classification, 32 patients (53.3%) had grade 4 ulcers, 16 patients (26.6%) had grade 5 ulcers, 10 patients (16.6%) had grade 3 ulcers and 2 patients (3.3%) had grade 2 ulcers. A one-to-one correspondence between the AI tool's recommended amputation level and the level actually performed was observed in 50 out of 60 cases (83.3%). In the remaining 10 cases, discrepancies were noted, with the AI consistently recommending a more proximal level of amputation than what was performed. The inter-rater agreement analysis between the actual surgeries and the AI tool's recommendations yielded a Cohen's kappa coefficient of 0.808 (SD: 0.055, 95% CI: 0.701-0.916), indicating substantial agreement. Relying solely on clinical photographs, ChatGPT-4.0 demonstrates decisions that are largely consistent with those of an experienced team in determining the optimal level of amputation for DFUs, with the exception of hindfoot amputations.

Attribute annotation and bias evaluation in visual datasets for autonomous driving

This paper addresses the often overlooked issue of fairness in the autonomous driving domain, particularly in vision-based perception and prediction systems, which play a pivotal role in the overall functioning of Autonomous Vehicles (AVs). We focus our analysis on biases present in some of the most commonly used visual datasets for training person and vehicle detection systems. We introduce an annotation methodology and a specialised annotation tool, both designed to annotate protected attributes of agents in visual datasets. We validate our methodology through an inter-rater agreement analysis and provide the distribution of attributes across all datasets. These include annotations for the attributes age, sex, skin tone, group, and means of transport for more than 90K people, as well as vehicle type, colour, and car type for over 50K vehicles. Generally, diversity is very low for most attributes, with some groups, such as children, wheelchair users, or personal mobility vehicle users, being extremely underrepresented in the analysed datasets. The study contributes significantly to efforts to consider fairness in the evaluation of perception and prediction systems for AVs. This paper follows reproducibility principles. The annotation tool, scripts and the annotated attributes can be accessed publicly at https://github.com/ec-jrc/humaint_annotator.

Large-scale pretrained frame generative model enables real-time low-dose DSA imaging: An AI system development and multi-center validation study.

Digital subtraction angiography (DSA) devices are commonly used in numerous interventional procedures across various parts of the body, necessitating multiple scans per procedure, which results in significant radiation exposure for both doctors and patients. Inspired by generative artificial intelligence techniques, this study proposes GenDSA, a large-scale pretrained multi-frame generative model-based real-time and low-dose DSA imaging system. GenDSA was developed to generate 1-, 2-, and 3-frame sequences following each real frame. A large-scale dataset comprising ∼3 million DSA images from 27,117 patients across 10 hospitals was constructed to pretrain, fine-tune, and validate GenDSA. Two other datasets from 25 hospitals were used for evaluation. Objective evaluations included SSIM and PSNR. Five interventional radiologists independently assessed the quality of the generated frames using the Likert scale and visual Turing test. Scoring consistency among the radiologists was measured using the Kendall coefficient of concordance (W). The Fleiss' kappa values were used for inter-rater agreement analysis for visual Turing tests. Using only one-third of the clinical radiation dose, videos generated by GenDSA were perfectly consistent with real videos. Objective evaluations demonstrated that GenDSA's performance (PSNR= 36.83, SSIM= 0.911, generation time= 0.07 s/frame) surpassed state-of-the-art algorithms. Subjective ratings and statistical results from five doctors indicated no significant difference between real and generated videos. Furthermore, the generated videos were comparable to real videos in overall quality (4.905 vs. 4.935) and lesion assessment (4.825 vs. 4.860). With clear clinical and translational values, the developed GenDSA can significantly reduce radiation damage to both doctors and patients during DSA-guided procedures. This study was supported by the National Key R&D Program and the National Natural Science Foundation of China.

Perspectives on the diagnosis and management of functional cognitive disorder: An international Delphi study.

Current proposed criteria for functional cognitive disorder (FCD) have not been externally validated. We sought to analyse the current perspectives of cognitive specialists in the diagnosis and management of FCD in comparison with neurodegenerative conditions. International experts in cognitive disorders were invited to assess seven illustrative clinical vignettes containing history and bedside characteristics alone. Participants assigned a probable diagnosis and selected the appropriate investigation and treatment. Qualitative, quantitative and inter-rater agreement analyses were undertaken. Eighteen diagnostic terminologies were assigned by 45 cognitive experts from 12 countries with a median of 13 years of experience, across the seven scenarios. Accurate discrimination between FCD and neurodegeneration was observed, independently of background and years of experience: 100% of the neurodegenerative vignettes were correctly classified and 75%-88% of the FCD diagnoses were attributed to non-neurodegenerative causes. There was <50% agreement in the terminology used for FCD, in comparison with 87%-92% agreement for neurodegenerative syndromes. Blood tests and neuropsychological evaluation were the leading diagnostic modalities for FCD. Diagnostic communication, psychotherapy and psychiatry referral were the main suggested management strategies in FCD. Our study demonstrates the feasibility of distinguishing between FCD and neurodegeneration based on relevant patient characteristics and history details. These characteristics need further validation and operationalisation. Heterogeneous labelling and framing pose clinical and research challenges reflecting a lack of agreement in the field. Careful consideration of FCD diagnosis is advised, particularly in the presence of comorbidities. This study informs future research on diagnostic tools and evidence-based interventions.

Can Artificial Intelligence Identify Reading Fluency and Level? Comparison of Human and Machine Performance

This study examined whether an artificial intelligence-based automatic speech recognition system can accurately assess students’ reading fluency and reading level. Participants were 120 fourth-grade students attending public schools in Türkiye. Students read a grade-level text out loud while their voice was recorded. Two experts and the artificial intelligence-based automatic speech recognition system analyzed the recordings for reading errors. Following the analysis, a word error rate was calculated for both the experts and the artificial intelligence-based automatic speech recognition system. Word error rates were converted into reading accuracy rate scores. Inter-rater agreement and linear regression analyses were used to compare the raters’ reading fluency scores, and logistic regression analyses were used to compare the classification of readers according to their reading levels. Results showed that the difference between the scores of the artificial intelligence-based automatic speech recognition system and the expert scores was minimal. This is because there was a very high level of agreement between the artificial intelligence-based automatic speech recognition system and the experts scores. Linear regression analyses showed that the artificial intelligence-based automatic speech recognition system significantly predicted the scores of experts. According to the logistic regression analysis results, the artificial intelligence-based automatic speech recognition system was at least 93% as successful as human raters in classifying readers as poor and good. These results give us hope that reading assessments at classroom, school, regional, national, and even international levels can be conducted more accurately and economically by using artificial intelligence-based systems in the coming years.

An automated flowchart for the Modified Rankin Scale assessment: A multicenter inter-rater agreement analysis.

The Modified Rankin Scale (mRS) is a widely adopted scale for assessing stroke recovery. Despite limitations, the mRS has been adopted as primary outcome in most recent clinical acute stroke trials. Designed to be used by multidisciplinary clinical staff, the congruency of this scale is not consistent, which may lead to mistakes in clinical or research application. We aimed to develop and validate an interactive and automated digital tool for assessing the mRS-the iRankin. A panel of five board-certified and mRS-trained vascular neurologists developed an automated flowchart based on current mRS literature. Two international experts were consulted on content and provided feedback on the prototype platform. The platform contained five vignettes and five real video cases, representing mRS grades 0-5. For validation, we invited neurological staff from six comprehensive stroke centers to complete an online assessment. Participants were randomized into two equal groups usual practice versus iRankin. The participants were randomly allocated in pairs for the congruency analysis. Weighted kappa (kw) and proportions were used to describe agreement. A total of 59 professionals completed the assessment. The kw was dramatically improved among nurses, 0.76 (95% confidence interval (CI) = 0.55-0.97) × 0.30 (0.07-0.67), and among vascular neurologists, 0.87 (0.72-1) × 0.82 (0.66-0.98). In the accuracy analysis, after the standard mRS values for the vignettes and videos were determined by a panel of experts, and considering each correct answer as equivalent to 1 point on a scale of 0-15, it revealed a higher mean of 10.6 (±2.2) in the iRankin group and 8.2 (±2.3) points in the control group (p = 0.02). In an adjusted analysis, the iRankin adoption was independently associated with the score of congruencies between reported and standard scores (beta coefficient = 2.22, 95% CI = 0.64-3.81, p = 0.007). The iRankin adoption led to a substantial or near-perfect agreement in all analyzed professional categories. More trials are needed to generalize our findings. Our user-friendly and free platform is available at https://www.irankinscale.com/.

Ultrafast Brain MRI with Deep Learning Reconstruction for Suspected Acute Ischemic Stroke.

Background Deep learning (DL)-accelerated MRI can substantially reduce examination times. However, studies prospectively evaluating the diagnostic performance of DL-accelerated MRI reconstructions in acute suspected stroke are lacking. Purpose To investigate the interchangeability of DL-accelerated MRI with conventional MRI in patients with suspected acute ischemic stroke at 1.5 T. Materials and Methods In this prospective study, 211 participants with suspected acute stroke underwent clinically indicated MRI at 1.5 T between June 2022 and March 2023. For each participant, conventional MRI (including T1-weighted, T2-weighted, T2*-weighted, T2 fluid-attenuated inversion-recovery, and diffusion-weighted imaging; 14 minutes 18 seconds) and DL-accelerated MRI (same sequences; 3 minutes 4 seconds) were performed. The primary end point was the interchangeability between conventional and DL-accelerated MRI for acute ischemic infarction detection. Secondary end points were interchangeability regarding the affected vascular territory and clinically relevant secondary findings (eg, microbleeds, neoplasm). Three readers evaluated the overall occurrence of acute ischemic stroke, affected vascular territory, clinically relevant secondary findings, overall image quality, and diagnostic confidence. For acute ischemic lesions, size and signal intensities were assessed. The margin for interchangeability was chosen as 5%. For interrater agreement analysis and interrater reliability analysis, multirater Fleiss κ and the intraclass correlation coefficient, respectively, was determined. Results The study sample consisted of 211 participants (mean age, 65 years ± 16 [SD]); 123 male and 88 female). Acute ischemic stroke was confirmed in 79 participants. Interchangeability was demonstrated for all primary and secondary end points. No individual equivalence indexes (IEIs) exceeded the interchangeability margin of 5% (IEI, -0.002 [90% CI: -0.007, 0.004]). Almost perfect interrater agreement was observed (P > .91). DL-accelerated MRI provided higher overall image quality (P < .001) and diagnostic confidence (P < .001). The signal properties of acute ischemic infarctions were similar in both techniques and demonstrated good to excellent interrater reliability (intraclass correlation coefficient, ≥0.8). Conclusion Despite being four times faster, DL-accelerated brain MRI was interchangeable with conventional MRI for acute ischemic lesion detection. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Haller in this issue.

Mixture polarization in inter-rater agreement analysis: a Bayesian nonparametric index

In several observational contexts where different raters evaluate a set of items, it is common to assume that all raters draw their scores from the same underlying distribution. However, a plenty of scientific works have evidenced the relevance of individual variability in different type of rating tasks. To address this issue the intra-class correlation coefficient (ICC) has been used as a measure of variability among raters within the Hierarchical Linear Models approach. A common distributional assumption in this setting is to specify hierarchical effects as independent and identically distributed from a normal with the mean parameter fixed to zero and unknown variance. The present work aims to overcome this strong assumption in the inter-rater agreement estimation by placing a Dirichlet Process Mixture over the hierarchical effects’ prior distribution. A new nonparametric index λ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\lambda$$\\end{document} is proposed to quantify raters polarization in presence of group heterogeneity. The model is applied on a set of simulated experiments and real world data. Possible future directions are discussed.

Assessment of inter-rater and intra-rater reliability of the Luna EMG robot as a tool for assessing upper limb proprioception in patients with stroke-a prospective observational study.

The aim of the study was to assess the inter-rater and intra-rater agreement of measurements performed with the Luna EMG (electromyography) multifunctional robot, a tool for evaluation of upper limb proprioception in individuals with stroke. The study was conducted in a group of patients with chronic stroke. A total of 126 patients participated in the study, including 78 women and 48 men, on average aged nearly 60 years (mean = 59.9). Proprioception measurements were performed using the Luna EMG diagnostic and rehabilitation robot to assess the left and right upper limbs. The examinations were conducted by two raters, twice, two weeks apart. The results were compared between the raters and the examinations. High consistency of the measurements performed for the right and the left hand was reflected by the interclass correlation coefficients (0.996-0.998 and 0.994-0.999, respectively) and by Pearson's linear correlation which was very high (r = 1.00) in all the cases for the right and the left hand in both the inter-rater and intra-rater agreement analyses. Measurements performed by the Luna EMG diagnostic and rehabilitation robot demonstrate high inter-rater and intra-rater agreement in the assessment of upper limb proprioception in patients with chronic stroke. The findings show that Luna EMG is a reliable tool enabling effective evaluation of upper limb proprioception post-stroke.

Validity of Ultrasound for the Diagnosis of Arterial Thoracic Outlet Syndrome

ObjectiveThoracic outlet syndrome (TOS) is a rare disorder mostly seen in younger individuals. Although patients’ wellbeing is relevantly impaired, it often takes a long time before the diagnosis is made. Digital subtraction angiography (DSA) is routinely used despite its radiation exposure, which is a major concern in this young patient population. Moreover, DSA offers limited opportunities for functional assessment. In contrast, ultrasonography is widely accessible without causing radiation exposure and allows for flexible functional assessment. The main goal of the study was to investigate whether ultrasound (US) is a viable alternative to DSA in diagnosing arterial TOS (aTOS). MethodsPatients referred to a tertiary centre for evaluation of suspected TOS were recruited into the study. DSA was routinely performed with the patient’s arms both in the raised (abducted) and neutral (adducted) position. Two vascular surgeons and two radiologists assessed the resulting images for presence of aTOS. Additionally, two examiners performed US according to a standardised protocol. The reference for presence of aTOS was the DSA based interdisciplinary vascular conference consensus. Inter-rater agreement and latent class analysis (LCA) were performed between assessors and diagnostic methods. ResultsFifty one consecutive patients (two thirds female) aged 39.3 ± 13.0 years were included within 11 months. US agreement was excellent at 0.94 (0.841 – 0.980), DSA agreement for vascular surgeons was good at 0.779 (0.479 – 1.000), whereas it was moderate at 0.546 (0.046 – 1.000) for radiologists. Results suggest that DSA is untenable as the gold standard for aTOS diagnosis. In LCA, US was shown to be a reliable diagnostic tool for the detection of aTOS. ConclusionUS examination is a valid test for the detection of haemodynamically relevant compression of arteries in the diagnostic work up of aTOS using a standardised protocol. The role of DSA as the gold standard should be reviewed and needs to be reconsidered.

Adaptação transcultural do inventário Parent Hearing Aid Management Inventory para o Português Brasileiro

ABSTRACT Purpose to translate and cross-culturally adapt the Parent Hearing Aid Management Inventory into Brazilian Portuguese. Methods study of the methodological type of cross-cultural adaptation, which followed the recommendations of the literature for its execution. Two steps and eight steps were performed to achieve the adaptation: obtaining permission from the authors; formation of a committee of specialists who acted in some of the steps for the validation of the translation, translation by 2 proficient translators, synthesis of the translations and evaluation of equivalences, reverse translation and synthesis of the same, pilot study with 10 families to verify the applicability of the instrument and synthesis of the final version of the instrument. Results Cohen's kappa analysis was applied for the inter-rater agreement analysis and Cronbach's alpha coefficient for the analysis of internal reliability of the instrument. After application with the families, the instrument was considered valid to assess the needs for guidance and support of families regarding the management of hearing devices in the population of children with hearing loss. Conclusion the Inventory was translated and adapted into Brazilian Portuguese, under the name of Inventário de Manejo dos Aprendidos for the Family (IMAAF) and has the potential to help in clinical practice to achieve effective use of individual sound amplification devices in the population of children with hearing impairment, in a perspective centered on the needs of their families.

Cross-cultural adaptation of the Parent Hearing Aid Management Inventory into Brazilian Portuguese.

to translate and cross-culturally adapt the Parent Hearing Aid Management Inventory into Brazilian Portuguese. study of the methodological type of cross-cultural adaptation, which followed the recommendations of the literature for its execution. Two steps and eight steps were performed to achieve the adaptation: obtaining permission from the authors; formation of a committee of specialists who acted in some of the steps for the validation of the translation, translation by 2 proficient translators, synthesis of the translations and evaluation of equivalences, reverse translation and synthesis of the same, pilot study with 10 families to verify the applicability of the instrument and synthesis of the final version of the instrument. Cohen's kappa analysis was applied for the inter-rater agreement analysis and Cronbach's alpha coefficient for the analysis of internal reliability of the instrument. After application with the families, the instrument was considered valid to assess the needs for guidance and support of families regarding the management of hearing devices in the population of children with hearing loss. the Inventory was translated and adapted into Brazilian Portuguese, under the name of Inventário de Manejo dos Aprendidos for the Family (IMAAF) and has the potential to help in clinical practice to achieve effective use of individual sound amplification devices in the population of children with hearing impairment, in a perspective centered on the needs of their families.

Modeling Bland–Altman Limits of Agreement with Fractional Polynomials—An Example with the Agatston Score for Coronary Calcification

Bland–Altman limits of agreement are very popular in method comparison studies on quantitative outcomes. However, a straightforward application of Bland–Altman analysis requires roughly normally distributed differences, a constant bias, and variance homogeneity across the measurement range. If one or more assumptions are violated, a variance-stabilizing transformation (e.g., natural logarithm, square root) may be sufficient before Bland–Altman analysis can be performed. Sometimes, fractional polynomial regression has been used when the choice of variance-stabilizing transformation was unclear and increasing variability in the differences was observed with increasing mean values. In this case, regressing the absolute differences on a function of the average and applying fractional polynomial regression to this end were previously proposed. This review revisits a previous inter-rater agreement analysis on the Agatston score for coronary calcification. We show the inappropriateness of a straightforward Bland–Altman analysis and briefly describe the nonparametric limits of agreement of the original investigation. We demonstrate the application of fractional polynomials, use the Stata packages fp and fp_select, and discuss the use of degree-2 (the default setting) and degree-3 fractional polynomials. Finally, we discuss conditions for evaluating the appropriateness of nonstandard limits of agreement.

Profound barriers to value management adoption on construction projects in a developing nation

Purpose The adoption of value management (VM) has been on the rise in the construction industry, especially in the past decade. Identifying the main barriers to VM application for ameliorating poor project outcomes in the construction industry is a critical step to overcoming its implementation. Therefore, this study aims to examine the barriers to the adoption of VM in the Nigerian construction industry (NCI) and identify the critical ones for action by construction stakeholders. Design/methodology/approach This study used two rounds of Delphi surveys with 15 VM experts to identify the challenges to VM adoption. The Delphi team’s data from the two surveys were analysed using descriptive statistics. The mean-mode approach was used to identify ten major barriers to VM adoption based on the consensus reached during the second round of the Delphi poll. Following that, an interrater agreement analysis and a significant level analysis were performed. Findings Of the 17 identified barriers hindering VM adoption on construction projects in the NCI, 10 were found to be profound after statistical analyses of the data. The significant barriers include communication gap between the client and other consultants; delay in design preparation because of VM services; the pretext that VM is being practiced in the construction industry; and the wrong perception of VM by construction professionals, among others. The findings of the analysis suggested that the lack of VM adoption in the NCI was because of inadequate knowledge of VM techniques among construction professionals. Therefore, it is advised that VM training be provided for concerned stakeholders to address the lack of VM expertise among Nigerian construction professionals. Practical implications This study investigated the barriers to VM adoption on construction projects in the NCI with the opinions of a few construction professionals who can be regarded as VM experts. As against obtaining large survey responses from construction professionals who may not be knowledgeable in the VM process, the limited VM experts’ responses provide a valid basis for drawing inferences and making recommendations to address the paucity of VM applications in the NCI. Originality/value Through a Delphi survey, the critical barriers to the adoption of VM were identified, along with recommendations to help construction stakeholders overcome the challenges and enjoy the benefits of VM.

Development of a Campus Facility Management Operational Framework Using a Modified Delphi Method

An effective facility management process is crucial to overall firm performance and is becoming a core competency for organizations seeking to gain a competitive advantage. The recent literature on facility management reveals that poor facility management is a major cause of management inefficiency, leading directly to budget overruns, maintenance delays, repetition of work, variations, noncompliance incidents, unnecessary risk, and dissatisfied customers. Therefore, facility management firms should pay close attention to the most critical success factors in order to effectively manage and measure performance. This paper presents a multidimensional campus facility management performance framework (CFMPF) that addresses both operational and systematic factors. Facility management companies, owners, contractors, and consultants can use this framework to identify systemic weaknesses using key indicators to specifically assess the effectiveness and success of facility management. The significance of 45 key facility management success factors across seven categories was determined via a literature review, expert interviews, and the Delphi method. The seven process groups are: Campus Facility Work Force Management, Campus Facility Communication Management, Campus Facility Systems Management, Campus Facility Organizational Management, Campus Facility Financial Management, Campus Sustainability & Environment Management, and Campus Facility Assets Management. To verify the overall agreement via the Delphi rounds, the score percentage and standard deviation to mean ratio were examined. Mean value and interrater agreement (IRA) analyses were conducted to evaluate the correlation between the factors and facility performance. This research adds to the body of knowledge by presenting a systematic operational framework for campus facility management that identifies the underlying factors influencing performance and fills the gaps left by previous studies.

Comparative analysis between the 8th and 7th AJCC as a prognostic tool for patients diagnosed with nonmetastatic head and neck soft-tissue sarcoma.

e23550 Background: Head and neck soft-tissue sarcoma (HN-STS) is a rare entity for which a new prognostic risk stratification has been proposed in the 8th AJCC. The tumor size 2cm and 4cm have been adopted as cutoffs for T1 to T3 classification and the invasion of surrounding structures denotes T4. In opposite, in the 7th edition the primary tumor size of 5 cm determines T1 and T2 classification. This study aimed to compare the 8th and 7th AJCC as a prognostic tool for HN-STS. Additionally, we sought to investigate prognostic factors in this group of patients. Methods: This is a single Tertiary Hospital study approved by the institutional ethical review committee from the AC Camargo Cancer Center. We retrospectively evaluated patients with non-metastatic HN-STS whose histology subtypes are included in the 40th chapter of the 8th AJCC version. We compared the AJCC 7th and 8th T staging with Cohen’s kappa coefficient using inter-rater agreement analysis. The Kaplan-Meier (KM) estimator was used to calculate median survival times; differences in time-to-event outcomes were assessed using the log-rank test. Cox proportional hazard model was used to assess the effect of features in overall survival (OS). Results: Ninety-five HN-STS treated with curative intent from 2005 from 2021 patients diagnosed were evaluated. Surgical resection (94.7%), radiotherapy (48.4%) and chemotherapy (23.1%) were the employed therapeutic approaches. Leiomyosarcoma and liposarcoma (10 of each subtype) were the most common histology subtypes; High grade disease was present in 66.3%. Kappa coefficient of inter-rater agreement was 0.46 with statistical significance (p = 0.03), which indicates poor agreement between AJCC 7th and 8th T staging system. Regarding OS stratified by T size in the AJCC 7th (T1 and T2) and 8th (T1, T2, T3 and T4) was not statistically different between groups (P = 0.83 and P = 0.21, respectively). Patients with T4 (47.6% vs 78.7%, P = 0.04) and high grade (64.0% vs 96.2%, P = 0.003) had worst 5-years OS. In Cox multivariate analysis, patients with 8th edition T4 and high grade had 3.15- and 4.16-fold increased risk of death, respectively. Conclusions: In our series of patients with HN-STS, neither 7th nor 8th AJCC T size category groups discriminated prognosis. Tumor with local invasion (8th AJCC T4) and high grade were all associated with increased risk of mortality.

526. Antimicrobial Selection Based on Precision Metagenomics Compared with Standard Urine Culture/Susceptibility: A Reliability and Inter-rater Agreement Analysis for Application in Patients with Urinary Tract Infections

Abstract Background Management of urinary tract infections (UTIs) is highly variable and ∼50% of affected women may receive inappropriate antimicrobials. Precision Metagenomics-based detection of pathogens and antimicrobial resistance (AMR) markers can inform antimicrobial selection in support of stewardship practices. We investigated inter-physician agreement of culture and metagenomic results in a retrospective analysis. Methods We used de-identified urine samples with results of urine culture &amp; antimicrobial susceptibility testing (AST, n=25, 76% culture-positive) and the Explify® Urinary Pathogen ID/AMR Panel (UPIP, IDbyDNA Inc). UPIP is a Precision Metagenomics method to detect &amp;gt;170 uropathogens and &amp;gt;2,000 AMR markers. Four physicians reviewed culture/AST and UPIP results independently to assess if and how they would treat based on each set of results in a middle-aged, female patient with uncomplicated UTI (no comorbidities or medication allergies). Consensus was defined by simple majority; treatment decisions and antimicrobial choice were adjudicated by an infectious disease trained pharmacist. Inter-rater agreement (Fleiss’ kappa) and reliability for accurate case management (reference: adjudicator) were estimated for each method. Results Analytical agreement for uropathogen identification between the two methods was 72% with 95% positive agreement for common uropathogens. Consensus on whether or not to treat was reached for 96% (24/25) of cases for both culture/AST and UPIP; decisions were concordant between methods for 88% of cases (22/25). Inter-rater agreement was high (culture: k=0.73, UPIP: k=0.68; p&amp;lt; 0.05). Physicians were more likely to decide to treat based on culture results (17/25, 68%) than UPIP (15/25, 60%). UPIP detected additional organism(s) and/or AMR marker(s) compared with culture/AST alone that were informative for stewardship-guided treatment decisions in 28% (7/25) of samples. Figure 1: Physician Consensus Treatment Decisions Based on Results of Urine Sample Analysis by Precision Metagenomics or Urine Culture/AST Four infectious disease physicians independently reviewed paired results from analysis of a set of 25 clinical remnant urine samples by an investigational Precision Metagenomics method (UPIP) and by standard urine culture/AST and determined whether or not treatment would be warranted. Figure 2:Summary of Changes in Antimicrobial Selection based on Interpretation of Precision Metagenomics vs Culture/AST Results An infectious disease pharmacist reviewed and adjudicated hypothetical antimicrobial selections. All antimicrobial selections based on the results of the Precision Metagenomics method (UPIP) were found to be appropriate; in 7/15 cases, hypothetical treatment based on UPIP results would have resulted in a change in management that was evaluated to be consistent with stewardship practices. Conclusion Physician agreement on hypothetical treatment recommendations was comparable between the investigational method (UPIP) and the reference method (urine culture/AST). In no case was there inappropriate consensus to treat based on UPIP results alone. The findings from this pilot analysis support the feasibility of metagenomics-guided management of UTIs. Disclosures Benjamin Briggs, MD, PhD, CloudCath: Advisor/Consultant|CosmosID: Employee Benjamin Briggs, MD, PhD, CloudCath: Advisor/Consultant|CosmosID: Employee Robert Schlaberg, MD, MPH, IDBYDNA INC: Board Member|IDBYDNA INC: Explify-related IP|IDBYDNA INC: Employee|IDBYDNA INC: Stocks/Bonds.

Exploring variations in current clinical practice regarding postoperative pericardial effusion: a questionnaire study

Abstract Background Postoperative pericardial effusion (PPE) is a frequently occurring event after cardiac surgery, potentially leading to life-threatening cardiac tamponade. Unfortunately, no guidelines are currently available on this subject, possibly leading to unwarranted variations in care in clinical practice. Purpose The goal of our study was to determine the diagnostic and therapeutic preferences of cardiologists and cardiothoracic surgeons in the Netherlands regarding PPE, and to evaluate possible variations in this matter between specialties and centers. Methods A web-based questionnaire was sent to all interventional cardiologists and cardiothoracic surgeons in the Netherlands. Diagnostic preferences for the evaluation of PPE were examined. Additionally, clinical preferences were explored in response to 4 patient scenarios with each illustrating a combination of either high or low clinical suspicion of cardiac tamponade and either high or low echocardiographic suspicion. Each scenario was also stratified by 3 different amounts of PPE (&amp;lt;1cm, 1–2cm, &amp;gt;2cm), leading to a total of 12 fictional patient cases (Figure 1). Results From 27 of the 31 contacted centers, a total of 94 clinicians responded to the survey, of whom 46 interventional cardiologists and 48 cardiothoracic surgeons. Cardiologists were more in favor of routine postoperative echocardiography in all patients (44%), whereas cardiothoracic surgeons preferred routine imaging after specific procedures, especially after mitral (85%) and tricuspid (79%) valve surgery. Overall, drainage of PPE (&amp;gt;48 hours postoperatively) was preferred through pericardiocentesis (83%) instead of surgical evacuation (17%). Regarding the patient scenarios, cardiothoracic surgeons were significantly more inclined to evacuate PPE compared to cardiologists for all scenarios combined (51% vs 37%, p&amp;lt;0.001) (Figure 1). In addition, evacuation was overall significantly more preferred by cardiologists employed in surgical centers when compared to those working in non-surgical centers (43% vs 31%, p=0.02). Results from the interrater agreement analysis of groups of clinicians working in the same center varied from very poor up to very good (kappa value of 0.22–0.67), suggesting existing variation of PPE treatment preferences even within one center. Conclusions Our study highlights the significant variation of preferences in contemporary PPE management between cardiothoracic surgeons and interventional cardiologists in the Netherlands, sometimes even within the same center. These findings are the possible consequences of a current lack of general recommendations or guidelines. This could lead to unwarranted variation in care and additional research is necessary to provide general recommendations. Funding Acknowledgement Type of funding sources: None.

Total serum vitamin B12 (cobalamin) LC-MS/MS assay as an arbiter of clinically discordant immunoassay results.

Measurement of the serum levels of vitamin B12 (VB12) is key for evaluating VB12 deficiency-dependent anemia. Immunoassay, the major method for determining VB12, tends to give false-normal results because of the presence of anti-intrinsic factor (IF-Ab) or other factors such as heterophilic antibodies etal. This study aimed to develop a liquid chromatography tandem mass spectrometry (LC-MS/MS) method that is helpful for distinguish false normal VB12 results measured by the immunoassay. Different forms of VB12 were derivatized into CN-B12, which was collected through solid-phase extraction and analyzed via LC-MS/MS. 236 serum samples were measured both by LC-MS/MS and immunoassay, results were compared, and the IF-Ab effect was evaluated. The LC-MS/MS assay afforded a linear slope from 20 to 4,000pmol/L for CN-B12. OH-VB12, methyl-VB12, and CoA-VB12 showed recovery within 89.3-109.5%. The intra-assay CV of VB12 was 2.6-4.1%, whereas the total CV was 9.3-9.8%. Passing-Bablok regression between LC-MS/MS and immunoassay results showed that the slope was 1.085 and the intercept was-15.691. The Bland-Altman plot showed that the mean difference and difference% were-34.6 pmol/L and 0.3%, respectively. Inter-rater agreement analysis showed that the linear weighted kappa value was 0.885, implying good agreement between the two methods. However, two samples were falsely elevated and one sample was falsely normal in the immunoassay compared with LC-MS/MS. The LC-MS/MS method helped in the distinction of false-normal VB12 results shown by the immunoassay. The VB12 LC-MS/MS method can be used as an arbiter of clinically discordant immunoassay results.