Level Of Inter-rater Agreement Research Articles

Assessment of carcinoembryonic antigen-related cell adhesion molecule 5 expression by immunohistochemistry in real-world clinical samples of non-small cell lung cancer.

e20013 Background: Carcinoembryonic antigen-related cell adhesion molecule 5 (CEACAM5) is an emerging target of antibody-drug conjugate therapy for non-small cell lung cancer (NSCLC). High expression of CEACAM5 has been reported in approximately 25% of patients with lung adenocarcinoma (Lefebvre AM, et al. Lung Cancer. 2023; 184:107356). However, CEACAM5 expression has not been systematically examined in a real-world patient population with NSCLC. In this study, we evaluated our experience in the assessment of CEACAM5 expression by immunohistochemistry (IHC) in routinely diagnosed clinical samples. Methods: We assessed the expression of CEACAM5 by IHC on clinical biopsy/resection samples of consecutively diagnosed NSCLC from September 2022 to June 2023 at our institution, using the anti-CEACAM5 clone 769 antibody assay protocol developed for tusamitamab ravtansine clinical trials. Only cases adequate for PD-L1 22C3 and Oncomine Comprehensive Assay v3 NGS assay were included. CEACAM5 expression was scored independently by three pathologists based on the method proposed by Lefebvre et al. Expression levels were categorized as high (≥ 50% tumor cells at ≥ 2+ intensity), moderate (1–49% tumor cells at ≥ 2+ intensity), and negative (0 or 1+ intensity). Interrater reliability was assessed by Kendall’s coefficient of concordance (KCC) and Fleiss’ Kappa. Discordant classification was reviewed with the final classification achieved by consensus. Association with PD-L1 TPS and NGS results was determined by Chi-squared test. Results: This study cohort consisted of 150 consecutively diagnosed NSCLC, including 133 adenocarcinomas and 17 squamous cell carcinomas. The samples were derived from both primary (n = 138) and metastatic sites (n = 12). Both membranous and cytoplasmic CEACAM5 expression was observed in neoplastic cells, but was consistently absent in nonneoplastic cells. The prevalence of consensus high CEACAM5 membranous expression was 21% (n = 32) overall, 20% in primary (n = 28) and 33% (n = 4) in metastatic sites. The level of interrater agreement across all categories was moderate (KCC = 0.77). Interrater agreement between high CEACAM5 expression versus moderate/negative categories combined was also moderate (Kappa = 0.60). Challenging features included the determination of staining intensity and mixed membranous and cytoplasmic staining pattern, and cases near the cut-offs of the defined categories. CEACAM5 expression was not associated with PD-L1 TPS (p = 0.42) or EGFR mutations (p = 0.44). Conclusions: Our data showed that approximately 20% of routinely diagnosed clinical NSCLC samples had high CEACAM5 expression by IHC. The interrater reliability on the determination of high CEACAM5 expression was moderate among the three pathologists. We highlighted the challenging aspects in the evaluation of this biomarker.

Reliability testing of the Health of the Nation Outcome Scales 2018.

The Health of the Nation Outcome Scales (HoNOS) is a widely used clinical measure designed to rate and monitor the outcomes of service users accessing specialist mental healthcare. Since its development (in 1996), numerous research studies have confirmed the HoNOS captures the aspects of care that it purports to (validity), and that clinicians' ratings are consistent both over time, and between different raters (reliability). In 2018, the HoNOS was reviewed with updates made to some terminology and other revisions intended to remove ambiguity in the guidance for raters. However, although the new version (HoNOS 2018) was accompanied by a recommendation that its validity and reliability be re-tested this was not undertaken. To our knowledge, this is the first study to re-assess the updated tool's reliability by measuring the level of agreement between different raters. Our findings confirm that there is an acceptable level of consistency between student mental health nurses that have been trained to use the (new) HoNOS 2018. The HoNOS is nationally mandated for use by all specialist mental healthcare providers in the UK. Our findings provide some assurance that, with appropriate update training and monitoring of organisational-level data sets, the original HoNOS glossary can safely be replaced with the HoNOS 2018 to ensure more contemporary routine outcome measurement can occur. INTRODUCTION: The Health of the Nation Outcome Scales (HoNOS) is a well-established clinician rated outcome measure for use in mental health services. Following an international review, an updated version (HoNOS 2018) was published with a recommendation that its psychometric properties be re-tested prior to widespread implementation. To date, only one such study has been published. To test the inter-rater agreement levels for HoNOS 2018. Third-year student mental health nurses received training to complete the HoNOS 2018. Following this timetabled session, they were each invited to independently rate two, randomly selected, videos of (simulated) patient interviews. The resulting data were then analysed to calculate the tool's internal consistency and inter-rater agreement levels. The 55 participants provided 106 ratings from four vignettes. Cronbach's alphas and McDonalds omegas confirmed the revised tool's internal consistency was acceptable. Average measure intraclass correlation coefficients for the four patient vignettes indicated excellent reliability. This study provides initial assurance that the HoNOS 2018 is a reliable clinician rated outcome measure suitable for use in routine clinical practice by relatively inexperienced mental health practitioners with limited training.

Diagnostic reliability and accuracy of the hydraulic contrast lift protocol in the radiographic detection of sinus lift and perforation: ex vivo randomized split-mouth study in an ovine model

ObjectivesAssessing the diagnostic reliability, validity, and accuracy of the hydraulic contrast lift protocol during transcrestal sinus floor elevation in detecting the lift and perforation of the sinus membrane before graft material application and assessing the effect of its use on the operator’s diagnostic confidence.Material and methodsA single-blind randomized split-mouth study on fresh refrigerated sheep heads. The first intervention consisted of injecting 0.5 ml iodinated contrast medium on the test side and 0.5 ml saline on the control side. In the second intervention artificial sinus membrane perforations were created followed by injecting 0.5 ml iodinated contrast medium on the test side and 0.5 ml saline on the control side. Intraoperative periapical radiographs were taken for both interventions. The resulting 40 radiographs were assessed by 10 examiners to provide interpretations and confidence ratings. The primary endpoints were diagnostic reliability, validity, accuracy, and perceived diagnostic confidence.ResultsIn the hydraulic contrast lift protocol, the detection rate was 99% for sinus elevations and 98% for perforations, the saline protocol yielded a detection rate of 28% and 20% respectively. The hydraulic contrast lift protocol demonstrated a high level of inter-rater agreement for the diagnosis of elevations (p < 0.001) and perforations (p < 0.001), strong diagnostic validity for the diagnosis of elevations (p < 0.001) and perforations (p < 0.001), high sensitivity and specificity (p < 0.001) and higher mean diagnostic confidence ratings for both interventions when compared to the saline protocol (p < 0.001). The difference between the predicted probability for correct diagnosis of the hydraulic contrast lift protocol and the saline protocol was significant (p < 0.001) for the detection of both elevations and perforations.ConclusionFollowing the hydraulic contrast lift protocol, the use of a radiographic contrast medium can reliably confirm sinus membrane lift and detect perforation during transcrestal sinus floor elevation prior to bone graft application in addition to improving the diagnostic confidence of the operator while relying on periapical radiographs.

Perceptual Evaluation of the Tracheoesophageal Voice: Italian Validation of the Sunderland Tracheoesophageal Perceptual Scale

This study aimed to validate the Sunderland Tracheoesophageal Perceptual Scale (SToPS) in the Italian language by checking the inter- and intra-rater reliability. Scale validation METHODS: The validation of the tool involved the following steps: (1) translation and adaptation of the SToPS into Italian language; (2) recruitment of participants (60 laryngectomized patients with a voice prosthesis and 12 health professionals-six speech and language therapists (SLTs) and six ear, nose, and throat surgeons-classified into experienced (Exp) or not at assessing voice; (3) recording of patients' speech samples; (4) perceptual evaluation of recorded speech samples (test and retest) performed by the 12 health professionals; and (5) statistical analysis (quadratic weighted Cohen kappa and weighted kappa of Light coefficients). When all raters were considered as one group, an agreement ≥0.50 was reached for all parameters. The "ExpSLTs" group rated more reliably than the other groups, achieving a "good" intrarater agreement for 9/10 parameters. Despite the interrater coefficients were lower than the intrarater ones for all rater groups, "ExpSLTs" obtained the best levels of interrater agreement, achieving a level of agreement ≥0.50 for all parameters. Finally, considering intrarater+interrater agreement ("good+good" or "good+moderate"), the "ExpSLTs" group showed the greatest agreement, attaining all parameters that met the criteria for "good" or "moderate" agreement levels. The Italian version of SToPS can be considered a reliable tool. As in the original version, expert SLTs are the better judges for tracheoesophageal voice assessment.

Intra- and inter-rater reliability of a manual codification system for footwear impressions: first lessons learned from the development of a footwear database for forensic intelligence purposes

To generate forensic intelligence from footwear impressions and link crime scenes, most law enforcement agencies and forensic laboratories rely on a manual codification system based on pattern recognition and classification by human analysts. However, although they are commonly used in practice, to date we still know little about the reliability of such systems. Taking advantage of the development of a footwear database for forensic intelligence purposes at the Laboratoire de sciences judiciaires et de médecine légale in Quebec (Canada), this study aims to make a preliminary assessment of the intra- and inter-rater reliability (i.e., the level of repeatability over time and the level of consensus between analysts) of the proposed codification system. To do so, three forensic intelligence analysts classified a set of 27 crime scene impressions and test impressions at two different times (two weeks apart). Percent agreement, Cohen’s Kappa, and Light’s Kappa were then calculated. Results show that two out of three analysts have reached an almost perfect level of intra-rater agreement, while the other have achieved a substantial level of intra-agreement, and that all analysts have reached a substantial level of inter-rater agreement. Findings suggest that, although a few patterns may have lower levels of agreement, overall, the developed codification system presents a satisfactory level of reliability. This preliminary study thus suggests that contrary to what advocates of fully automated systems may sometimes imply, manual codification of footwear impressions may be fairly appropriate for intelligence purposes. It calls for further evaluative research in the field.

Inter-observer agreement on levels of evidence in radiology articles

Levels of evidence (LE) are established through a hierarchical classification of studies according to their design. At present, there are many heterogeneous LE classifications, and this hampers their applicability. Our study aims to identify which LE classification has the best interobserver concordance for radiology articles. For this purpose, an interobserver agreement analysis were performed on 105 original articles applying two NE scales (Oxford Center of Evidence Based Medicine (OCEBM) y National Health and Medical Research Council (NHMRC)). The inter-rater agreement of the LE assigned after reading the abstracts was good when using the OCEBM scale (K = 0.679), and somewhat lower with the NHMRC (K = 0.577 -moderate-). All differences were statistically significant (P < .000). So, in conclusion, of the two scales analysed (OCEBM and NHMRC), the OCEBM led to the strongest level of inter-rater agreement.

Concordancia interobservador de los niveles de evidencia en artículos radiológicos

Levels of evidence (LE) are established through a hierarchical classification of studies according to their design. At present, there are many heterogeneous LE classifications, and this hampers their applicability. Our study aims to identify which LE classification has the best interobserver concordance for radiology articles. For this purpose, an interobserver agreement analysis were performed on 105 original articles applying two NE scales (Oxford Center of Evidence Based Medicine (OCEBM) y National Health and Medical Research Council (NHMRC)). The inter-rater agreement of the LE assigned after reading the abstracts was good when using the OCEBM scale (K=0.679), and somewhat lower with the NHMRC (K=0.577 -moderate-). All differences were statistically significant (p<0.000). So, in conclusion, of the two scales analysed (OCEBM and NHMRC), the OCEBM led to the strongest level of inter-rater agreement.

Field Reliability of Forensic Mental Health Evaluations of Offenders for Post-Prison Civil Commitment

ABSTRACT This study examined the field reliability of forensic mental health examiners’ opinions in pre-parole evaluations. The California Offenders with Mental Health Disorders (OMD) statute allows for the post-incarceration commitment to a state hospital for treatment as a condition of parole. Reliability is critical in these evaluations because the triers of fact rely on them for their determinations. We expect to find variability among opinions, not only on the final opinions but also on each criterion. We compared three expert opinions on 98 cases to determine the level of interrater agreement. Agreement rates varied by agency, evaluator, and the forensic question posed. We found higher reliability for criteria that were more reliant on past factors and lower reliability for those criteria based on factors present at the time of the assessment. Imprecisely defined legal constructs, unknown individual evaluator characteristics, contextual factors, and passage of time between the evaluations impede reliability.

Automated Vibration and Acoustic Crepitus Sensing in Humans

Abstract Crepitus vibrational and acoustic signal analysis of the human facet joints of the lumbar spine has historically been a difficult problem due to the inhomogeneous and varied signal characteristics. Although previous studies have examined tribological characteristics of facet joints, we have found vibrational and acoustic signals to be a satisfactory analog to facet joint tribology and function. Here, we improve upon our previous automated computational method, now enhancing it for the analysis of human crepitus. Compared with this group’s previous studies using a mechanical model, human crepitus is extremely complex. Therefore, we proposed an automated method (AM) of analysis that used a test set (n = 16) and an experimental set of data (n = 48). We had a fair level of interrater agreement (Kw = 0.367, standard error (SE) = 0.054, 95% confidence interval (CI) = 0.260–0.474) between the AM and human observers before adjustments were made in the AM. Following adjustments to the AM, interrater agreement improved to a substantial level (Kw = 0.788, SE = 0.056, 95% CI = 0.0.682–0.895). In the future, we recommend a machine learning study with a larger number of subjects who can better capture the nuances of varying types of human crepitus.

The Positive Predictive Value of Pediatric Stroke Diagnoses in Administrative Data: A Retrospective Validation Study.

This retrospective cohort study aimed to examine the positive predictive value (PPV) of pediatric stroke diagnoses in the Danish National Registry of Patients (DNRP) and the impact of different stroke definitions on the PPV. We included children registered with a stroke or stroke-related diagnosis in the DNRP between January 2017 through December 2020. Two assessors reviewed medical records and validated cases according to the American Heart and American Stroke Association (AHA/ASA) stroke definition. The level of interrater agreement was examined using kappa statistics. Validation by the AHA/ASA definition was compared with validation according to the definition in the International Classification of Disease 11th version (ICD-11) and the World Health Organization's definition. Stroke was confirmed in 120 of 309 included children, yielding an overall PPV of 0.39 (95% CI: 0.33-0.45). PPV varied across stroke subtypes from 0.83 (95% CI: 0.71-0.92) for ischemic stroke (AIS), 0.57 (95% CI: 0.37-0.76) for unspecified stroke, 0.42 (95% CI: 0.33-0.52) for intracerebral hemorrhage (ICH) to 0.31 (95% CI: 0.55-0.98) and 0.07 (95% CI: 0.01-0.22) for cerebral venous thrombosis and subarachnoid hemorrhage (SAH), respectively. Most non-confirmed ICH and SAH diagnoses were in children with traumatic intracranial hemorrhages (36 and 66% respectively). Among 70 confirmed AIS cases, 25 (36%) were identified in non-AIS code groups. PPV varied significantly across stroke definitions with the highest for the AHA/ASA definition (PPV = 0.39, 95% CI: 0.34-0.45) and the lowest for the WHO definition (PPV = 0.29, 95% CI: 0.24-0.34). Correspondingly, the incidence of pediatric AIS per 100.000 person-years changed from 1.5 for the AHA/ASA definition to 1.2 for ICD-11 and 1.0 for the WHO-definition. The overall interrater agreement was considered excellent (κ=0.85). After validation, stroke was confirmed in only half of the children registered in the DNRP with a stroke-specific diagnosis. Non-validated administrative data should be used with caution in pediatric stroke research. Pediatric stroke incidence rates may vary markedly depending on which stroke definition is used.

Improved Image Caption Rating - Datasets, Game, and Model.

How well a caption fits an image can be difficult to assess due to the subjective nature of caption quality. What is a good caption? We investigate this problem by focusing on image-caption ratings and by generating high quality datasets from human feedback with gamification. We validate the datasets by showing a higher level of inter-rater agreement, and by using them to train custom machine learning models to predict new ratings. Our approach outperforms previous metrics - the resulting datasets are more easily learned and are of higher quality than other currently available datasets for image-caption rating.

Interrater Reliability of Functional Lumen Imaging Probe Panometry and High-Resolution Manometry for the Assessment of Esophageal Motility Disorders.

High-resolution manometry (HRM) and functional lumen imaging probe (FLIP) are primary and/or complementary diagnostic tools for the evaluation of esophageal motility. We aimed to assess the interrater agreement and accuracy of HRM and FLIP interpretations. Esophageal motility specialists from multiple institutions completed the interpretation of 40 consecutive HRM and 40 FLIP studies. Interrater agreement was assessed using intraclass correlation coefficient (ICC) for continuous variables and Fleiss' κ statistics for nominal variables. Accuracies of rater interpretation were assessed using the consensus of 3 experienced raters as the reference standard. Fifteen raters completed the HRM and FLIP studies. An excellent interrater agreement was seen in supine median integral relaxation pressure (ICC 0.96, 95% confidence interval 0.95-0.98), and a good agreement was seen with the assessment of esophagogastric junction (EGJ) outflow, peristalsis, and assignment of a Chicago Classification version 4.0 diagnosis using HRM (κ = 0.71, 0.75, and 0.70, respectively). An excellent interrater agreement for EGJ distensibility index and maximum diameter (0.91 [0.90-0.94], 0.92 [0.89-0.95]) was seen, and a moderate-to-good agreement was seen in the assignment of EGJ opening classification, contractile response pattern, and motility classification (κ = 0.68, 0.56, and 0.59, respectively) on FLIP. Rater accuracy for Chicago Classification version 4.0 diagnosis on HRM was 82% (95% confidence interval 78%-84%) and for motility diagnosis on FLIP Panometry was 78% (95% confidence interval 72%-81%). Our study demonstrates high levels of interrater agreement and accuracy in the interpretation of HRM and FLIP metrics and moderate-to-high levels for motility classification in FLIP, supporting the use of these approaches for primary or complementary evaluation of esophageal motility disorders.

Distinguishing lipid-poor angiomyolipoma from renal carcinoma using tumor shape.

To validate the "overflowing beer sign" (OBS) for distinguishing between lipid-poor angiomyolipoma (AML) and renal cell carcinoma, and to determine whether it improves the detection of lipid-poor AML when added to the angular interface sign, a previously-validated morphologic feature associated with AML. Retrospective nested case-control study of all 134 AMLs in an institutional renal mass database matched 1:2 with 268 malignant renal masses from the same database. Cross-sectional imaging from each mass was reviewed and the presence of each sign was identified. A random selection of 60 masses (30 AML and 30 benign) was used to measure interobserver agreement. Both signs were strongly associated with AML in the total population (OBS: OR 17.4 95% CI 8.0-42.5, p < 0.001; angular interface: OR 12.6, 95% CI 5.9-29.7, p < 0.001) and the population of patients excluding those with visible macroscopic fat (OBS: OR 11.2, 95% CI 4.8-28.7, p < 0.001; angular interface: 8.5, 95% CI 3.7-21.1, p < 0.001). In the lipid-poor population, the specificity of both signs was excellent (OBS: 95.6%, 95% CI 91.9%-98%; angular interface: 95.1%, 95% CI 91.3%-97.6%). Sensitivity was low for both signs (OBS: 31.4%, 95% CI 24.0-45.4%; angular interface: 30.5%, 95% CI 20.8%-41.6%). Both signs showed high levels of inter-rater agreement (OBS 90.0% 95% CI 80.5 - 95.9; angular interface 88.6, 95% CI 78.7-94.9) Testing for AML using the presence of either sign in this population improved sensitivity (39.0%, 95% CI 28.4%-50.4%, p = 0.023) without significantly reducing specificity (94.2%, 95% CI 90%-97%, p = 0.2) relative to the angular interface sign alone. Recognition of the OBS increases the sensitivity of detection of lipid-poor AML without significantly reducing specificity.

Reporting accuracy of pressure injury categorisation in an acute tertiary hospital: A four-year analysis.

To determine the reporting accuracy of pressure injury categorisation by bedside clinicians, compared with nurse experts. Pressure injuries are an enduring complication of hospitalisation. The categorisation of pressureinjury affects treatment and management decision-making and use of resources, and severe hospital-acquired pressureinjury incidence is used to benchmark quality of care. However, it is unclear how accurately pressureinjuries are categorised by clinicians in practice. Secondary analysis of hospital pressure injury incident and validation data. All pressure injuries reported in adults between 2016 and 2019 that were subsequently validated by nurse experts were analysed. Absolute agreement is reported using percentages, with inter-rater agreement reported using Kappa measure of agreement. The GRRAS reporting guideline was followed. Of 6186 pressureinjuries that were analysed, the category was reported correctly in 67.3% (n= 4163), with an overall moderate level of inter-rater agreement by category (Κ= .567, p< .001). Of those found to be non-pressure injuries when validated (18.3%, n= 1129), most were reported originally as stage II (41.2%, n= 465) or stage I (30.5%, n= 344), and 13.4% (n= 151) were categorised initially as unstageable. The majority reported initially as stage I, stage II, suspected deep tissue injury or mucosal pressure injury were validated, whereas half of those reported initially as stage III or IV were validated and less than a third of those reported initially as unstageable pressure injuries were validated. This study provides important insight into the accuracy of pressure injury categorisation. Whilst moderate agreement of categorisation was found between reporting clinicians and nurse experts, pressure injury differential diagnosis and categorisation of severe injuries were inadequate. These results may be used for benchmarking and provide a focal point for future education and practice improvement efforts. Neither patients nor the public were directly involved in the project.

Interrater reliability for the detection of cortical lesions on phase-sensitive inversion recovery magnetic resonance imaging in patients with multiple sclerosis.

To assess the reliability of phase-sensitive inversion recovery (PSIR) magnetic resonance imaging (MRI) and its accuracy for determining the topography of demyelinating cortical lesions in patients with multiple sclerosis (MS). This was a cross-sectional study conducted at a tertiary referral center for MS and other demyelinating disorders. We assessed the agreement among three raters for the detection and topographic classification of cortical lesions on fluid-attenuated inversion recovery (FLAIR) and PSIR sequences in patients with MS. We recruited 71 patients with MS. The PSIR sequences detected 50% more lesions than did the FLAIR sequences. For detecting cortical lesions, the level of interrater agreement was satisfactory, with a mean free-response kappa (κFR) coefficient of 0.60, whereas the mean κFR for the topographic reclassification of the lesions was 0.57. On PSIR sequences, the raters reclassified 366 lesions (20% of the lesions detected on FLAIR sequences), with excellent interrater agreement. There was a significant correlation between the total number of lesions detected on PSIR sequences and the Expanded Disability Status Scale score (ρ = 0.35; p < 0.001). It seems that PSIR sequences perform better than do FLAIR sequences, with clinically satisfactory interrater agreement, for the detection and topographic classification of cortical lesions. In our sample of patients with MS, the PSIR MRI findings were significantly associated with the disability status, which could influence decisions regarding the treatment of such patients.

RESEARCH ON VOICE QUALITY IN CALL CENTER DISCOURSE: VALIDATING RELIABILITY THROUGH INTERRATER AGREEMENT AND PRAAT

Voice quality is an important component of effective communication in call center conversations. A qualitative research methodology was used in the present study to examine how Filipino customer service representatives (CSRs) and American customers use voice quality features in call center discourse. A precise evaluation of voice quality features can aid in the development of effective communication strategies. Through interrater agreement, the present study aimed to validate the reliability of voice quality feature assessment. In this study, how Filipino CSRs and American customers' call center telephone conversations differed in terms of voice quality characteristics such as volume (Loud/Soft), pitch (High/Low), tension (Tense/Lax), and rhythm (Fast/Slow) was investigated. The conversation transcripts were examined using the aforementioned voice quality features. The transcriptions were then evaluated by three independent raters to determine interrater agreement. The findings revealed a high level of interrater agreement between the three raters of all voice quality features, with up to 0.8 in identifying voice quality changes in generic stages. In particular, a higher agreement was found in the assessment of specific voice quality features such as loudness, high pitch, and tension. Praat software was also used to aid in the analysis of some voice recordings to validate the interrater agreement reliability. The approach taken in the present study for assessing voice quality using interrater agreement and Praat software can improve service quality in the call center industry and provides a reliable and valid framework for future qualitative research in applied linguistics.

Detection and diagnosis of posterior circulation calcification: An interobserver agreement study

BackgroundPosterior cerebral ischemic stroke and its underlying causes can be easily misdiagnosed in routine practice. Therefore, more than a third of positive cases can be easily missed during routine CT image reporting unless expert neuroimaging radiologists carefully report it. ObjectiveTo assess the inter-rater agreement level between senior residents and general radiologists and a specialized expert neuroradiologist when detecting and diagnosing posterior cerebral circulation calcification. MethodsThis was a cross-sectional observational study. A total of fifteen senior radiology residents (SRRs) and fifteen general radiologists (GRs) at four different hospitals in Riyadh, Saudi Arabia, were included in the study. A total of four CT-scanned brain cases with the presence of posterior circulation calcification (PCC) with different degrees of severity and one brain case with negative PCC were selected. These cases were predefined by expert neuroradiologists at our center. The cases were uploaded into the picture archiving and communication systems (PACS) at four different centers as outsider cases. These cases were then randomly assigned to the participating SRRs and GRs for reporting. All radiologists were blinded to the findings of the cases. Inter-observer agreement was assessed using the weighted kappa coefficient of agreement (k) between the two groups. ResultsThe cerebral calcification misdiagnosis rate for the SRRs and GRs was > 93% for most of the positive cases. There was 1) poor inter-observer agreement between the SRRs and GRs for the detection of severe posterior cerebral calcification(PCC) in a negative stroke case (agreement for misdiagnosis, k = 0.93; correct diagnosis, k = 0.00), 2) poor inter-observer agreement for mild PCC in a negative stroke case (agreement for misdiagnosis, k = 0.93; correct diagnosis, k = 0.00), 3) moderate PCC in a positive posterior stroke case (agreement for misdiagnosis, k = 0.92; correct diagnosis, k = 0.00), and 4) poor interobserver agreement for severe PCC in a positive posterior cerebral stroke case (agreement for misdiagnosis, k = 0.846; correct diagnosis, k = 0.00). There was excellent agreement between the SRRs and GRs when reporting negative cases of PCC and stroke. ConclusionOur study concluded that most of the SRRs and GRs missed the diagnosis of posterior cerebral calcification in the presented cases.

Reliability of COVID-19 data: An evaluation and reflection.

The rapid proliferation of COVID-19 has left governments scrambling, and several data aggregators are now assisting in the reporting of county cases and deaths. The different variables affecting reporting (e.g., time delays in reporting) necessitates a well-documented reliability study examining the data methods and discussion of possible causes of differences between aggregators. To statistically evaluate the reliability of COVID-19 data across aggregators using case fatality rate (CFR) estimates and reliability statistics. Cases and deaths were collected daily by volunteers via state and local health departments, as primary sources and newspaper reports, as secondary sources. In an effort to begin comparison for reliability statistical analysis, BroadStreet collected data from other COVID-19 aggregator sources, including USAFacts, Johns Hopkins University, New York Times, The COVID Tracking Project. COVID-19 cases and death counts at the county and state levels. Lower levels of inter-rater agreement were observed across aggregators associated with the number of deaths, which manifested itself in state level Bayesian estimates of COVID-19 fatality rates. A national, publicly available data set is needed for current and future disease outbreaks and improved reliability in reporting.

A Robust, Highly Multiplexed Mass Spectrometry Assay to Identify SARS-CoV-2 Variants.

ABSTRACTSevere acute respiratory syndrome coronavirus 2 (SARS-CoV-2) variants are characterized by differences in transmissibility and response to therapeutics. Therefore, discriminating among them is vital for surveillance, infection prevention, and patient care. While whole-genome sequencing (WGS) is the “gold standard” for variant identification, molecular variant panels have become increasingly available. Most, however, are based on limited targets and have not undergone comprehensive evaluation. We assessed the diagnostic performance of the highly multiplexed Agena MassARRAY SARS-CoV-2 Variant Panel v3 to identify variants in a diverse set of 391 SARS-CoV-2 clinical RNA specimens collected across our health systems in New York City, USA and Bogotá, Colombia (September 2, 2020 to March 2, 2022). We demonstrated almost perfect levels of interrater agreement between this assay and WGS for 9 of 11 variant calls (κ ≥ 0.856) and 25 of 30 targets (κ ≥ 0.820) tested on the panel. The assay had a high diagnostic sensitivity (≥93.67%) for contemporary variants (e.g., Iota, Alpha, Delta, and Omicron [BA.1 sublineage]) and a high diagnostic specificity for all 11 variants (≥96.15%) and all 30 targets (≥94.34%) tested. Moreover, we highlighted distinct target patterns that could be utilized to identify variants not yet defined on the panel, including the Omicron BA.2 and other sublineages. These findings exemplified the power of highly multiplexed diagnostic panels to accurately call variants and the potential for target result signatures to elucidate new ones.IMPORTANCE The continued circulation of SARS-CoV-2 amid limited surveillance efforts and inconsistent vaccination of populations has resulted in the emergence of variants that uniquely impact public health systems. Thus, in conjunction with functional and clinical studies, continuous detection and identification are quintessential to informing diagnostic and public health measures. Furthermore, until WGS becomes more accessible in the clinical microbiology laboratory, the ideal assay for identifying variants must be robust, provide high resolution, and be adaptable to the evolving nature of viruses like SARS-CoV-2. Here, we highlighted the diagnostic capabilities of a highly multiplexed commercial assay to identify diverse SARS-CoV-2 lineages that circulated from September 2, 2020 to March 2, 2022 among patients seeking care in our health systems. This assay demonstrated variant-specific signatures of nucleotide/amino acid polymorphisms and underscored its utility for the detection of contemporary and emerging SARS-CoV-2 variants of concern.

Getting on the Same Page: The Impact of Interviewer Education and Structured Interviews on Interrater Agreement in Residency Interviews

We explored the impact of implementing structured interviews and associated interviewer education on interrater agreement within a large academic residency program. Faculty and senior resident interviewers from a large academic residency program participated in a 3-hour structured interview course. Before and after the course, participants completed a 15-item assessment pertaining to the characteristics, logistics, and guidelines associated with structured interviews. Along with interviewer training, interview day logistics also changed from an unstructured format (no specific questions, one overall 1-9 rating scale) to a structured interview format, including incorporation of behavioral-based competency questions that would be asked of every applicant and behavioral anchored rating scales (1-10; 10 = highest). Interrater agreement was assessed via intraclass correlation coefficients (ICC1) for the 2 years before and 2 years after incorporation of the structured interview format. A total of 45 faculty and resident interviewers participated in the course in 2018. Participant knowledge significantly increased from an average of 36% to 79% after the course (p < 0.01). Prior to the intervention, overall interrater agreement was "poor" to "fair," with an ICC1 of 0.51 in 2016 and 0.49 in 2017. After the structured interview intervention, overall agreement increased to the "good" level with an ICC1 of 0.71 in 2018 and 0.66 in 2019. The proportion of applicants who received interview scores with at least 2 ratings more than 2 points apart significantly decreased from 59% to 47% after the intervention (p < 0.01). Incorporating an interviewer educational session and a structured interview format into residency selection can help increase agreement in ratings between interviewers. However, these data suggest that ongoing refresher trainings may be needed to maintain acceptable levels of interrater agreement.