Gwet's Agreement Coefficient Research Articles

Using ChatGPT to Provide Patient-Specific Answers to Parental Questions in the PICU.

To determine if ChatGPT can incorporate patient-specific information to provide high-quality answers to parental questions in the PICU. We hypothesized that ChatGPT would generate high-quality, patient-specific responses. In this cross-sectional study, we generated assessments and plans for 3 PICU patients with respiratory failure, septic shock, and status epilepticus and paired them with 8 typical parental questions. We prompted ChatGPT with instructions, an assessment and plan, and 1 question. Six PICU physicians evaluated the responses for accuracy (1-6), completeness (yes/no), empathy (1-6), and understandability (Patient Education Materials Assessment Tool, PEMAT, 0% to 100%; Flesch-Kincaid grade level). We compared answer quality among scenarios and question types using the Kruskal-Wallis and Fischer's exact tests. We used percent agreement, Cohen's Kappa, and Gwet's agreement coefficient to estimate inter-rater reliability. All answers incorporated patient details, utilizing them for reasoning in 59% of sentences. Responses had high accuracy (median 5.0, [interquartile range (IQR), 4.0-6.0]), empathy (median 5.0, [IQR, 5.0-6.0]), completeness (97% of all questions), and understandability (PEMAT % median 100, [IQR, 87.5-100]; Flesch-Kincaid level 8.7). Only 4/144 reviewer scores were <4/6 in accuracy, and no response was deemed likely to cause harm. There was no difference in accuracy, completeness, empathy, or understandability among scenarios or question types. We found fair, substantial, and almost perfect agreement among reviewers for accuracy, empathy, and understandability, respectively. ChatGPT used patient-specific information to provide high-quality answers to parental questions in PICU clinical scenarios.

MRI-based Zero Echo Time and Black Bone Pseudo-CT Compared with Whole-Body CT to Detect Osteolytic Lesions in Multiple Myeloma.

Background MRI is highly sensitive for assessing bone marrow involvement in multiple myeloma (MM) but does not enable detection of osteolysis. Purpose To assess the diagnostic accuracy, repeatability, and reproducibility of pseudo-CT MRI sequences (zero echo time [ZTE], gradient-echo black bone [BB]) in detecting osteolytic lesions in MM using whole-body CT as the reference standard. Materials and Methods In this prospective study, consecutive patients were enrolled in our academic hospital between June 2021 and December 2022. Inclusion criteria were newly diagnosed MM, monoclonal gammopathy of undetermined significance at high risk for MM, or suspicion of progressive MM. Participants underwent ZTE and BB sequences covering the lumbar spine, pelvis, and proximal femurs as part of 3-T whole-body MRI examinations, as well as clinically indicated fluorine 18 fluorodeoxyglucose PET/CT examination within 1 month that included optimized whole-body CT. Ten bone regions and two scores (categorical score = presence/absence of osteolytic lesion; semiquantitative score = osteolytic lesion count) were assessed by three radiologists (two experienced and one unfamiliar with pseudo-CT reading) on the ZTE, BB, and whole-body CT images. The accuracy, repeatability, and reproducibility of categorical scores (according to Gwet agreement coefficients AC1 and AC2) and differences in semiquantitative scores were assessed at the per-sequence, per-region, and per-patient levels. Results A total of 47 participants (mean age, 67 years ± 11 [SD]; 27 male) were included. In experienced readers, BB and ZTE had the same high accuracy (98%) in the per-patient analysis, while BB accuracy ranged 83%-100% and ZTE accuracy ranged 74%-94% in the per-region analysis. An increase of false-negative (FN) findings in the spine ranging from +17% up to +23%, according to the lumbar vertebra, was observed using ZTE (P < .013). Regardless of the region (except coxal bones), differences in the BB score minus the ZTE score were positively skewed (P < .021). Regardless of the sequence or region, repeatability was very good (AC1 ≥0.87 for all), while reproducibility was at least good (AC2 ≥0.63 for all). Conclusion Both MRI-based ZTE and BB pseudo-CT sequences of the lumbar spine, pelvis, and femurs demonstrated high diagnostic accuracy in detecting osteolytic lesions in MM. Compared with BB, the ZTE sequence yielded more FN findings in the spine. ClinicalTrials.gov Identifier: NCT05381077 Published under a CC BY 4.0 license. Supplemental material is available for this article.

A New Method for Documenting Sign Language Productions in Schools

Purpose: There are well-established guidelines for the recording, transcription, and analysis of spontaneous oral language samples by researchers, educators, and speech pathologists. In contrast, there is presently no consensus regarding methods for the written documentation of sign language samples. The Handshape Analysis Recording Tool (HART) is an innovative method for documenting and analyzing word level samples of signed languages in real time. Fluent sign language users can document the expressive sign productions of children to gather data on sign use and accuracy. Method: The HART was developed to document children's productions in Australian Sign Language (Auslan) in a bilingual–bicultural educational program for the Deaf in Australia. This written method was piloted with a group of fluent signing Deaf educational staff in 2014–2016, then used in 2022–2023 with a group of fluent signing professionals to examine inter- and intrarater reliability when coding parameters of sign accuracy. Results: Interrater reliability measured by Gwet's Agreement Coefficient, was “good” to “very good” across the four phonological parameters that are components of every sign: location, movement, handshape, and orientation. Conclusions: The findings of this study indicate that the HART can be a reliable tool for coding the accuracy of location, orientation, movement, and handshape parameters of Auslan phonology when used by professionals fluent in Auslan. The HART can be utilized with any sign language to gather word level sign language samples in a written form and document the phonological accuracy of signed productions.

Preoperative CT improves the assessment of stability in trochanteric hip fractures.

To investigate if preoperative CT improves detection of unstable trochanteric hip fractures. A single-centre prospective study was conducted. Patients aged 65 years or older with trochanteric hip fractures admitted to Stavanger University Hospital (Stavanger, Norway) were consecutively included from September 2020 to January 2022. Radiographs and CT images of the fractures were obtained, and surgeons made individual assessments of the fractures based on these. The assessment was conducted according to a systematic protocol including three classification systems (AO/Orthopaedic Trauma Association (OTA), Evans Jensen (EVJ), and Nakano) and questions addressing specific fracture patterns. An expert group provided a gold-standard assessment based on the CT images. Sensitivities and specificities of surgeons' assessments were estimated and compared in regression models with correlations for the same patients. Intra- and inter-rater reliability were presented as Cohen's kappa and Gwet's agreement coefficient (AC1). We included 120 fractures in 119 patients. Compared to radiographs, CT increased the sensitivity of detecting unstable trochanteric fractures from 63% to 70% (p = 0.028) and from 70% to 76% (p = 0.004) using AO/OTA and EVJ, respectively. Compared to radiographs alone, CT increased the sensitivity of detecting a large posterolateral trochanter major fragment or a comminuted trochanter major fragment from 63% to 76% (p = 0.002) and from 38% to 55% (p < 0.001), respectively. CT improved intra-rater reliability for stability assessment using EVJ (AC1 0.68 to 0.78; p = 0.049) and for detecting a large posterolateral trochanter major fragment (AC1 0.42 to 0.57; p = 0.031). A preoperative CT of trochanteric fractures increased detection of unstable fractures using the AO/OTA and EVJ classification systems. Compared to radiographs, CT improved intra-rater reliability when assessing fracture stability and detecting large posterolateral trochanter major fragments.

A qualitative examination of weapon violence against teachers: A theoretical framework and analysis.

Weapon violence in schools is a pressing concern with serious consequences. In this study, we propose and evaluate a theoretical framework of school-based weapon violence comprised of contributors, triggers, and motivation leading to weapon behaviors, taking into account weapon type, origin, and availability. This framework provides a foundation to investigate the multifaceted nature of weapon violence in schools. Specifically, we examine the weapon violence experiences of 417 U.S. teachers based on their reports of their most upsetting experiences with violence in their schools from various aggressors (i.e., students, parents, colleagues). Qualitative open-ended survey data were coded in NVivo after achieving strong interrater reliability (Gwet's agreement coefficient with first-order chance correction, AC₁ = .97; κ = .80), and analyses were guided by the proposed theoretical framework. Results indicated that individual, school, peer, family, and community conditions contributed to situational triggers (teacher or other school-stakeholder actions), and aggressor motivation was typically instrumental or expressive. The type and origin of weapons also played a role in weapon behaviors of carrying, threats, and usage. Aggressors often used readily available objects (e.g., chair, pencil) as weapons against teachers in addition to traditional weapons (e.g., knives, guns). Findings suggest that weapon violence in schools requires a broader conceptualization beyond traditional weapons and violence between students. This study advances our understanding of pathways to weapon behaviors for prevention and intervention. Implications of findings for school-stakeholder training and policies are discussed. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Looking Beyond Mortality Prediction: Primary Care Physician Views of Patients' Palliative Care Needs Predicted by a Machine Learning Tool.

To assess primary care physicians' (PCPs) perception of the need for serious illness conversations (SIC) or other palliative care interventions in patients flagged by a machine learning tool for high 1-year mortality risk. We surveyed PCPs from four Brigham and Women's Hospital primary care practice sites. Multiple mortality prediction algorithms were ensembled to assess adult patients of these PCPs who were either enrolled in the hospital's integrated care management program or had one of several chronic conditions. The patients were classified as high or low risk of 1-year mortality. A blinded survey had PCPs evaluate these patients for palliative care needs. We measured PCP and machine learning tool agreement regarding patients' need for an SIC/elevated risk of mortality. Of 66 PCPs, 20 (30.3%) participated in the survey. Out of 312 patients evaluated, 60.6% were female, with a mean (standard deviation [SD]) age of 69.3 (17.5) years, and a mean (SD) Charlson Comorbidity Index of 2.80 (2.89). The machine learning tool identified 162 (51.9%) patients as high risk. Excluding deceased or unfamiliar patients, PCPs felt that an SIC was appropriate for 179 patients; the machine learning tool flagged 123 of these patients as high risk (68.7% concordance). For 105 patients whom PCPs deemed SIC unnecessary, the tool classified 83 as low risk (79.1% concordance). There was substantial agreement between PCPs and the tool (Gwet's agreement coefficient of 0.640). A machine learning mortality prediction tool offers promise as a clinical decision aid, helping clinicians pinpoint patients needing palliative care interventions.

Validity evidence for a team-leading assessment tool in pediatric emergency resuscitations using video review.

Effective leadership of health care action teams has demonstrated positive influence on team performance and patient care, but there is no consensus on how to assess these skills. We developed a novel team leadership assessment tool for leaders of interprofessional pediatric resuscitation teams and collected validity evidence for this tool using video review. This was a prospective cohort study from November 2021 to October 2022. A novel team leadership assessment tool was developed using literature review and local expertise and then piloted and refined using medical simulation. Pediatric emergency medicine (PEM) fellows from a single tertiary care pediatric medical center were enrolled, and videos of one medical resuscitation and one trauma resuscitation were collected per fellow each month. Three reviewers underwent reviewer training and then scored the videos using the assessment tool. Raters provided feedback on feasibility and ease of use using a 5-point Likert scale. Inter-rater reliability for the assessment tool using Gwet's agreement coefficient and the association between performance and clinical level of training using generalized linear mixed model were calculated. Twelve PEM fellows enrolled and 146 videos were reviewed. The inter-rater reliability for each domain ranged from 0.45 (p < 0.0001) to 0.59 (p < 0.0001), with the inter-rater reliability of the total score being 0.49 (p < 0.0001). The reviewers' mean ratings of the elements of the tool were as follows: clarity of the domains (4.6/5), the independence of each domain from each other (3.9/5), the ease of use of the 5-point Likert scale (4.5/5), the usefulness of the provided examples for each domain (4.6/5), and the ability to assess each domain without having to rewatch (4.5/5). The tool differentiated between levels of clinical training for two of the six domains (p < 0.02). We developed a novel team leadership assessment tool for pediatric resuscitation team leaders that demonstrated moderate inter-rater reliability. The tool was easy to use and feasible for educators to assess the performance of PEM trainees in complex high-stakes clinical situations.

Inter-rater reliability and agreement of the General Movement Assessment and Motor Optimality Score-Revised in a large population-based sample

BackgroundPrechtl's General Movement Assessment (GMA) at fidgety age (3–5 months) is a widely used tool for early detection of cerebral palsy. Further to GMA classification, detailed assessment of movement patterns at fidgety age is conducted with the Motor Optimality Score-Revised (MOS-R). Inter-rater reliability and agreement are properties that inform test application and interpretation in clinical and research settings. This study aims to establish the inter-rater reliability and agreement of the GMA classification and MOS-R in a large population-based sample. MethodsA cross-sectional study of 773 infants from birth-cohort in Perth, Western Australia. GMA was conducted on home-recorded videos collected between 12 + 0 and 16 + 6 weeks post term age. Videos were independently scored by two masked experienced assessors. Inter-rater reliability and agreement were assessed using intraclass correlation coefficient and limits of agreement respectively for continuous variables, and Cohen's Kappa and Gwet's Agreement Coefficient, and percentage agreement respectively for discrete variables. ResultsThe classification of GMA showed almost perfect reliability (AC1 = 0.999) and agreement (99.9 %). Total MOS-R scores showed good-excellent reliability (ICC 0.857, 95 % CI 0.838–0.876) and clinically acceptable agreement (95 % limits of agreement of ±2.5 points). Substantial to almost perfect reliability and agreement were found for all MOS-R domain subscores. While MOS-R domains with higher redundancy in their categorisation have higher reliability and agreement, inter-rater reliability and agreement are substantial to almost perfect at the item level and are consistent across domains. ConclusionGMA at fidgety age shows clinically acceptable inter-rater reliability and agreement for GMA classification and MOS-R for population-based cohorts assessed by experienced assessors.

Assessment of enthesopathy in conventional pelvis radiography: Another tool for spondyloarthritis diagnosis beyond sacroiliitis

Introduction/objectivesEnthesitis is a cardinal feature of spondylarthritis (SpA), and the pelvis is a common site of enthesitis. This study aimed to establish the association between pelvic enthesis involvement on pelvic X-ray and SpA diagnosis through a radiographic enthesis index (REI) and to assess the reliability and accuracy of this REI. Materials and methodsThe participants were SpA patients and a control group composed of patients with chronic lumbar pain without SpA. Three blinded observers assessed each pelvic radiography three times. Three zones were used: Zone I (ZI), the iliopubic ramus; Zone II (ZII), the pubic symphysis, and Zone III (ZIII), the ischiopubic ramus. A grading system was created from 0 to 3 [Grade 0, normal; Grade 1, minimal changes (subcortical bone demineralization and/or periosteal wishkering, seen as radiolucency and trabeculation of the cortical bone upon tendon insertion); Grade 2, destructive changes (Grade 1 findings and erosions at the enthesis site); and Grade 3, findings of Grade 2 plus >2 mm whiskering out of the cortical bone) for the REI. The sum of the results of the three zones was called the total REI. For statistical analysis, we used the weighted kappa statistic adjusted for prevalence and bias using Gwet's agreement coefficient. ResultsWe enrolled 161 patients, 111 of them with SpA (39.6 % with axial SpA and 47.7 % with peripheral SpA) and 50 without SpA. In the SpA group, 36.7 % and 25.7 % had REI Grades 2 and 3 in ZIII, respectively, while only 6 % of the controls had these grades. For ZI, the frequency of Grades 1 to 3 was 42.3 % in the SpA group (8.1 %, 14.4 %, and 19.8 %, respectively), compared to only 2 % in the controls. ZII was unaffected in most of the patients with SpA (82.9 %) and in the controls (98 %). In the control group, Grade 0 was the most common REI grade in all three zones. The agreement was almost perfect for each zone and between the independent readers. The ROC-curve analysis showed that the highest performance areas were the total REI, ZIII, and ZI. Most (75 %) of the SpA patients without sacroiliitis on X-ray were REI-positive. The sensitivity of the REI for SpA diagnosis was 82 %, while the sensitivity of sacroiliitis on X-ray was 38.7 %. ConclusionsThe assessment of pelvic enthesis using the REI on pelvic radiography may be useful for SpA diagnosis. Total REI, ZIII, and ZI had the highest accuracy and almost perfect reliability. The REI is especially helpful in patients without sacroiliitis on imaging.

BI-RADS Category Assignments by GPT-3.5, GPT-4, and Google Bard: A Multilanguage Study.

Background The performance of publicly available large language models (LLMs) remains unclear for complex clinical tasks. Purpose To evaluate the agreement between human readers and LLMs for Breast Imaging Reporting and Data System (BI-RADS) categories assigned based on breast imaging reports written in three languages and to assess the impact of discordant category assignments on clinical management. Materials and Methods This retrospective study included reports for women who underwent MRI, mammography, and/or US for breast cancer screening or diagnostic purposes at three referral centers. Reports with findings categorized as BI-RADS 1-5 and written in Italian, English, or Dutch were collected between January 2000 and October 2023. Board-certified breast radiologists and the LLMs GPT-3.5 and GPT-4 (OpenAI) and Bard, now called Gemini (Google), assigned BI-RADS categories using only the findings described by the original radiologists. Agreement between human readers and LLMs for BI-RADS categories was assessed using the Gwet agreement coefficient (AC1 value). Frequencies were calculated for changes in BI-RADS category assignments that would affect clinical management (ie, BI-RADS 0 vs BI-RADS 1 or 2 vs BI-RADS 3 vs BI-RADS 4 or 5) and compared using the McNemar test. Results Across 2400 reports, agreement between the original and reviewing radiologists was almost perfect (AC1 = 0.91), while agreement between the original radiologists and GPT-4, GPT-3.5, and Bard was moderate (AC1 = 0.52, 0.48, and 0.42, respectively). Across human readers and LLMs, differences were observed in the frequency of BI-RADS category upgrades or downgrades that would result in changed clinical management (118 of 2400 [4.9%] for human readers, 611 of 2400 [25.5%] for Bard, 573 of 2400 [23.9%] for GPT-3.5, and 435 of 2400 [18.1%] for GPT-4; P < .001) and that would negatively impact clinical management (37 of 2400 [1.5%] for human readers, 435 of 2400 [18.1%] for Bard, 344 of 2400 [14.3%] for GPT-3.5, and 255 of 2400 [10.6%] for GPT-4; P < .001). Conclusion LLMs achieved moderate agreement with human reader-assigned BI-RADS categories across reports written in three languages but also yielded a high percentage of discordant BI-RADS categories that would negatively impact clinical management. © RSNA, 2024 Supplemental material is available for this article.

Performance of VI-RADS in predicting muscle-invasive bladder cancer after transurethral resection: a single center retrospective analysis.

To assess VIRADS performance and inter-reader agreement for detecting muscle-invasive bladder cancer (MIBC) following transurethral resection of bladder tumor (TURBT). An IRB-approved, HIPAA-compliant, retrospective study from 2016 to 2020 included patients with bladder urothelial carcinoma who underwent MRI after TURBT, and cystectomy within 3months without post-MRI treatments. Three radiologists blinded to pathology results independently reviewed MR images and assigned a VI-RADS score. Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and accuracy of VI-RADS were assessed for diagnosing MIBC using VI-RADS scores ≥ 3 and ≥ 4. Inter-reader agreement was assessed using Gwet's agreement coefficient (AC) and percent agreement. The cohort consisted of 70 patients (mean age, 68years ± 11 [SD]; range 39-85; 58 men) and included 32/70 (46%) with MIBC at cystectomy. ROC analysis revealed an AUC ranging from 0.67 to 0.77 and no pairwise statistical difference between readers (p-values, 0.06, 0.08, 0.97). Percent sensitivity, specificity, PPV, NPV and accuracy for diagnosing MIBC for the three readers ranged from 81.3-93.8, 36.8-55.3, 55.6-60.5, 77.3-87.5, and 62.9-67.1 respectively for VI-RADS score ≥ 3, and 78.1-81.3, 47.4-68.4, 55.6-67.6, 72.0-78.8 and 61.4-72.9 respectively for VI-RADS score ≥ 4. Gwet's AC was 0.63 [95% confidence interval (CI): 0.49,0.78] for VI-RADS score ≥ 3 with 79% agreement [95% CI 72,87] and 0.54 [95%CI 0.38,0.70] for VI-RADS score ≥ 4 with 76% agreement [95% CI 69,84]. VIRADS performance was not statistically different among 31/70 (44%) patients who received treatments prior to MRI (p ≥ 0.16). VI-RADS had moderate sensitivity and accuracy but low specificity for detection of MIBC following TURBT, with moderate inter-reader agreement.

High Peritumoral and Intratumoral T2 Signal Intensity in HER2-Positive Breast Cancers on Preneoadjuvant Breast MRI: Assessment of Associations With Histopathologic Characteristics.

BACKGROUND. Intratumoral necrosis and peritumoral edema are features of aggressive breast cancer that may present as high T2 signal intensity (T2 SI). Implications of high T2 SI in HER2-positive cancers are unclear. OBJECTIVE. The purpose of this study was to assess associations with histopathologic characteristics of high peritumoral T2 SI and intratumoral T2 SI of HER2-positive breast cancer on MRI performed before initiation of neoadjuvant therapy. METHODS. This retrospective study included 210 patients (age, 24-82 years) with 211 HER2 breast cancers who, from January 1, 2015, to July 30, 2022, underwent breast MRI before receiving neoadjuvant therapy. Two radiologists independently assessed cancers for high peritumoral T2 SI and high intratumoral T2 SI on fat-suppressed T2-weighted imaging and classified patterns of high peritumoral T2 SI (adjacent to tumor vs prepectoral extension). A third radiologist resolved discrepancies. Multivariable logistic regression analyses were performed to identify associations of high peritumoral and intratumoral T2 SI with histopathologic characteristics (associated ductal carcinoma in situ, hormone receptor status, histologic grade, lymphovascular invasion, and axillary lymph node metastasis). RESULTS. Of 211 HER2-positive cancers, 81 (38.4%) had high peritumoral T2 SI, and 95 (45.0%) had high intratumoral T2 SI. A histologic grade of 3 was independently associated with high peritumoral T2 SI (OR = 1.90; p = .04). Otherwise, none of the five assessed histopathologic characteristics were independently associated with high intratumoral T2 SI or high peritumoral T2 SI (p > .05). Cancers with high T2 SI adjacent to the tumor (n = 29) and cancers with high T2 SI with prepectoral extension (n = 52) showed no significant difference in frequency for any of the histopathologic characteristics (p > .05). Sensitivities and specificities for predicting the histopathologic characteristics ranged from 35.6% to 43.7% and from 59.7% to 70.7%, respectively, for high peritumoral T2 SI, and from 37.3% to 49.6% and from 49.3% to 62.7%, respectively, for high intratumoral T2 SI. Interreader agreement was almost perfect for high peritumoral T2 SI (Gwet agreement coefficient [AC] = 0.93), high intratumoral T2 SI (Gwet AC = 0.89), and a pattern of high peritumoral T2 SI (Gwet AC = 0.95). CONCLUSION. The only independent association between histopathologic characteristics and high T2 SI of HER2-positive breast cancer was observed between a histologic grade of 3 and high peritumoral T2 SI. CLINICAL IMPACT. In contrast with previously reported findings in broader breast cancer subtypes, peritumoral and intratumoral T2 SI had overall limited utility as prognostic markers of HER2-positive breast cancer.

A simple, clinically usable whole-body MRI system of joint assessment in adolescents and young people with juvenile idiopathic arthritis.

To introduce and evaluate a simple method for assessing joint inflammation and structural damage on whole-body MRI (WBMRI) in juvenile idiopathic arthritis (JIA), which is usable in clinical practice. The proposed system utilizes post-contrast Dixon WBMRI scans. Joints are assessed for synovitis (grade 0-2) and structural damage (present/absent) at 81 sites. The synovitis grading is based on features including above-normal intensity synovial enhancement, synovial hypertrophy, joint effusion, subarticular bone marrow oedema and peri-articular soft tissue oedema.This system was evaluated in a prospective study of 60 young people (47 patients with JIA and 13 controls with non-inflammatory musculoskeletal pain) who underwent a WBMRI. Three readers (blinded to diagnosis) independently reviewed all images and re-reviewed 20 individual scans. The intra- and inter-reader overall agreement (OA) and the intra- and inter-reader Gwet's agreement coefficients 2 (GAC2) were measured for the detection of a) participants with ≥1 joint with inflammation or structural damage and b) joint inflammation or structural damage for each joint. The inter-reader OA for detecting patients with ≥1 joint with inflammation, defined as grade 2 synovitis (G2), and ≥1 joint with structural damage were 80% and 73%, respectively. The intra-reader OA for readers 1-3 was 80-90% and 75-90%, respectively. The inter-reader OA and GAC2 for joint inflammation (G2) at each joint were both ≥85% for all joints but were lower if grade 1 synovitis was included as positive. The intra- and inter-reader agreements of this WBMRI assessment system are adequate for assessing objective joint inflammation and damage in JIA.

Adapting the Client Priority Rating Scale to better fit the sexual health counselling setting: a quality improvement study.

When demand for counselling in community-based clinics exceeds capacity, waiting lists are typically formed. Determining allocation priority solely on wait time can overlook client risk factors that can elevate priority. We undertook to rigorously adapt the only existing validated counselling triage tool, to better fit the sexual health setting. Sexual health counsellors were surveyed about aspects of client presentations that flagged increased priority. The revised Client Priority Rating Scale (CPRS-R) was created through systematic analysis and decision making, informed by survey results and literature review. Four expert sexual health counsellors independently rated the priority of 14 hypothetical clinical vignettes using the CPRS and CPRS-R. Criterion (concurrent), content and face validity are evidenced in the revised scale. Average interrater agreement was higher on the CPRS-R (28%) than the CPRS (11%); however, this difference was marginal (P =0.06). According to Gwet's Agreement Coefficient (AC) and Krippendorff's Alpha, both the CPRS and the CPRS-R demonstrate comparable interrater reliability, substantial and moderate, respectively. Kendall's W indicates the CPRS yielded higher reliability. However, the difference is not substantial. The CPRS-R is a triage tool designed for the sexual health counselling setting. This tool has demonstrated criterion, content and face validity, as well as moderate to substantial inter-rater reliability. It can be used in sexual health settings to inform assessments about client priority, along with clinical judgement and peer consultation.

Clinical Adjudication of Hemodialysis Catheter-Related Bloodstream Infections: Findings from the REDUCCTION Trial.

Hemodialysis central venous catheter-related bloodstream infections (HD-CRBSI) are a significant source of morbidity and mortality among dialysis patients, but benchmarking remains difficult due to varying definitions of HD-CRBSI. This study explored the effect of clinical adjudication process on HD-CRBSI reporting. The REDUcing the burden of Catheter ComplicaTIOns: a National approach (REDUCCTION) trial implemented an evidence-based intervention bundle using a stepped-wedge design to reduce HD-CRBSI rates in 37 Australian kidney services. Six New Zealand services participated in an observational capacity. Adult patients with a new hemodialysis catheter between December 2016 and March 2020 were included. HD-CRBSI events reported were compared to the adjudicated outcomes using the endpoint definition and adjudication processes of the REDUCCTION trial. The concordance level was estimated using Gwet's agreement coefficient (AC1) adjusted for service-level effects and implementation tranches (Australia only), with the primary outcome being the concordance of confirmed HD-CRBSI. A total of 744 hemodialysis catheter-related infectious events were reported among 7,258 patients, 12,630 catheters and 1.3 million catheter-exposure days. The majority were confirmed HD-CRBSI, with 77.9% agreement and 'substantial' concordance (AC1=0.77, 95% confidence interval [CI]: 0.73, 0.81). Exit site infections have the highest concordance (AC1=0.85, 95% CI 0.78, 0.91); the greatest discordance was in events classified as Other (AC1=0.33, 95% CI 0.16, 0.49). The concordance of all hemodialysis catheter infectious events remained 'substantial' (AC1=0.80, 95% CI 0.76, 0.83) even after adjusting for the intervention tranches in Australia and overall service-level clustering. There was a substantial level of concordance in overall and service-level reporting of confirmed HD-CRBSI. A standardized endpoint definition of HD-CRBSI resulted in comparable hemodialysis catheter infection rates in Australian and New Zealand kidney services. Consistent endpoint definition could enable reliable benchmarking outside clinical trials without the need for independent clinical adjudication.

Evaluation of a Novel Veterinary Dental Radiography Artificial Intelligence Software Program.

There is a growing trend of artificial intelligence (AI) applications in veterinary medicine, with the potential to assist veterinarians in clinical decisions. A commercially available, AI-based software program (AISP) for detecting common radiographic dental pathologies in dogs and cats was assessed for agreement with two human evaluators. Furcation bone loss, periapical lucency, resorptive lesion, retained tooth root, attachment (alveolar bone) loss and tooth fracture were assessed. The AISP does not attempt to diagnose or provide treatment recommendations, nor has it been trained to identify other types of radiographic pathology. Inter-rater reliability for detecting pathologies was measured by absolute percent agreement and Gwet's agreement coefficient. There was good to excellent inter-rater reliability among all raters, suggesting the AISP performs similarly at detecting the specified pathologies compared to human evaluators. Sensitivity and specificity for the AISP were assessed using human evaluators as the reference standard. The results revealed a trend of low sensitivity and high specificity, suggesting the AISP may produce a high rate of false negatives and may not be a good tool for initial screening. However, the low rate of false positives produced by the AISP suggests it may be beneficial as a "second set of eyes" because if it detects the specific pathology, there is a high likelihood that the pathology is present. With an understanding of the AISP, as an aid and not a substitute for veterinarians, the technology may increase dental radiography utilization and diagnostic potential.

An Investigation of Inter-Rater and Intra-Proxy Agreement in Measuring Quality of Life of Children in the Community Using the EQ-5D-Y-3L.

Self-reporting of health-related quality of life (HRQoL) in children is not always feasible. To date, proxy perspectives (Proxy versions 1 and 2) using the EQ-5D-Y-3L have not been explored for its impact on agreement with child self-report. Proxy version 1 requires the proxy to consider their own view of the child's HRQoL (proxy-proxy), while with Proxy version 2, the proxy is asked to respond as they believe their child would self-report their HRQoL (proxy-child). This study compared the inter-rater and intra-proxy agreement (overall and dimension level) using the EQ-5D-Y-3L self, proxy-proxy, and proxy-child reports. A community-based sample of child (aged 6-12 years) and parent dyads were invited to participate in a semi-structured interview. The child self-completed the EQ-5D-Y-3L independently of the parent who completed the EQ-5D-Y-3L from proxy-proxy and proxy-child perspectives. Agreement was determined using Concordance Correlation Coefficients (CCCs) for the overall (preference-weighted) HRQoL, while agreement at the dimension level was evaluated using Gwet's agreement coefficient (AC1). To assess the differences between the self and the two proxy reports, the Wilcoxon matched-pair signed-rank test was used. This study involved 85 child-parent dyads. The agreement between self and proxy overall HRQoL was low (fair) with both proxy-proxy (CCC=0.28) and proxy-child (CCC=0.26) reports. The largest discrepancy in the child-proxy agreement at dimension level with both the proxy versions was observed for 'feeling worried, sad or unhappy'. Within this dimension, the proxy-child perspective resulted in a stronger agreement (AC1=0.7, good) with child self-report compared with the traditional proxy-proxy perspective (AC1=0.58, moderate). Although the preference-weighted HRQoL was consistent across both the proxy perspectives, a significant difference was observed in the EQ VAS scores (p=0.02). This study demonstrates that choice of proxy perspective may have an impact on the problems reported on HRQoL dimensions and EQ VAS scores. However, in this community-based sample of generally healthy children, no significant difference was observed in the inter-rater agreement for child-self and proxy preference-weighted EQ-5D-Y-3L values based on proxy perspectives. While this suggests that preference-weighted data are not sensitive to the choice of perspective, these findings may differ for different HRQoL instruments and for alternative value sets with different properties.

Thin-slice 2D MR Imaging of the Shoulder Joint Using Denoising Deep Learning Reconstruction Provides Higher Image Quality Than 3D MR Imaging

This study was conducted to evaluate whether thin-slice 2D fat-saturated proton density-weighted images of the shoulder joint in three imaging planes combined with parallel imaging, partial Fourier technique, and denoising approach with deep learning-based reconstruction (dDLR) are more useful than 3D fat-saturated proton density multi-planar voxel images. Eighteen patients who underwent MRI of the shoulder joint at 3T were enrolled. The denoising effect of dDLR in 2D was evaluated using coefficient of variation (CV). Qualitative evaluation of anatomical structures, noise, and artifacts in 2D after dDLR and 3D was performed by two radiologists using a five-point Likert scale. All were analyzed statistically. Gwet's agreement coefficients were also calculated. The CV of 2D after dDLR was significantly lower than that before dDLR (P < 0.05). Both radiologists rated 2D higher than 3D for all anatomical structures and noise (P < 0.05), except for artifacts. Both Gwet's agreement coefficients of anatomical structures, noise, and artifacts in 2D and 3D produced nearly perfect agreement between the two radiologists. The evaluation of 2D tended to be more reproducible than 3D. 2D with parallel imaging, partial Fourier technique, and dDLR was proved to be superior to 3D for depicting shoulder joint structures with lower noise.

Variation observed in consensus judgments between pairs of reviewers when assessing the risk of bias due to missing evidence in a sample of published meta-analyses of nutrition research

ObjectivesTo evaluate the risk of bias due to missing evidence in a sample of published meta-analyses of nutrition research using the Risk Of Bias due to Missing Evidence (ROB-ME) tool and determine inter-rater agreement in assessments. Study Design and SettingWe assembled a random sample of 42 meta-analyses of nutrition research. Eight assessors were randomly assigned to one of four pairs. Each pair assessed 21 randomly assigned meta-analyses, and each meta-analysis was assessed by two pairs. We calculated raw percentage agreement and chance corrected agreement using Gwet's Agreement Coefficient (AC) in consensus judgments between pairs. ResultsAcross the eight signaling questions in the ROB-ME tool, raw percentage agreement ranged from 52% to 100%, and Gwet's AC ranged from 0.39 to 0.76. For the risk-of-bias judgment, the raw percentage agreement was 76% (95% confidence interval 60% to 92%) and Gwet's AC was 0.47 (95% confidence interval 0.14 to 0.80). In seven (17%) meta-analyses, either one or both pairs judged the risk of bias due to missing evidence as “low risk”. ConclusionOur findings indicated substantial variation in assessments in consensus judgments between pairs for the signaling questions and overall risk-of-bias judgments. More tutorials and training are needed to help researchers apply the ROB-ME tool more consistently.

Programmed death ligand-1 (PD-L1) clone 22C3 expression in resected colorectal cancer as companion diagnostics for immune checkpoint inhibitor therapy: A comparison study and inter-rater agreement evaluation across proposed cut-offs and predictive (TPS, CPS and IC) scores

BackgroundExpression of programmed death ligand-1 (PD-L1) guides the use of immune checkpoint inhibitors (ICI) in several cancers. In colorectal cancer (CRC), ICI are only approved for metastatic CRC, while several studies suggest high efficacy even in operable CRC. The aim of this study was to investigate the inter-rater agreement of PD-L1 as a companion diagnostic marker. MethodsSpecimens from resected stage I-III CRC (n = 166 tumors) were stained with PD-L1 22C3 clone. PD-L1 expression was scored by two pathologists as tumor proportion score (TPS), combined positive score (CPS) and immune cell score (IC). Inter-rater agreement was tested using three different agreement coefficients. ResultsRaw scores of the two pathologists had ‘good’ to ‘excellent’ correlation. Spearman's rho for TPS=0.917 (95 %CI 0.839–0.995), for CPS=0.776 (95 %CI 0.726–0.826) and IC=0.818 (95 %CI 0.761–0.875). For TPS, kappa (κ)-agreements for both the ≥1 % and ≥10 % cutoffs had excellent correlation. For CPS the ≥1 % and ≥10 % cutoffs demonstrated κ=0.32 (95 %CI 0.12–0.51) and κ=0.36 (95 %CI 0.25–0.48) respectively. Cutoffs for IC showed κ=0.53 (95 %CI 0.18–0.79) for the ≥1 % cutoff, and κ=0.61 (95 %CI 0.48–0.73) for the ≥10 % cutoff. Gwet's agreement coefficient (AC1) showed higher agreement coefficients than κ-values for most, but not all cut-offs. ConclusionAgreement for PD-L1 was good to excellent for raw scores. Agreement variation across several criteria and cut-offs suggests the need for more robust criteria for PD-L1 as a companion diagnostic marker.