Inter-rater Research Articles

Facial trauma education in radiology: using surgeon feedback as the benchmark for success.

Interpreting CT studies of facial trauma is challenging, and there are often substantial differences in the characterization of complex facial trauma between radiologists and surgeons. We designed a collaborative multidisciplinary project to reconcile differences in facial fracture interpretation through an educational intervention. The effectiveness of this intervention was evaluated through surgeon feedback on radiology reports. Radiology residents, neuroradiology fellows, and neuroradiology attendings were recruited as participants at a single tertiary care academic center. Otolaryngology residents were recruited as evaluators. Participants completed surveys and provided preliminary reports for example cases of facial trauma before and after attending an educational session. Evaluators performed a blinded review of these preliminary reports based on ideal reports developed by surgical and neuroradiology attendings. 26 participants (20 residents, 1 neuroradiology fellow, 5 neuroradiology attendings) completed the study. Six otolaryngology residents participated as evaluators. To assess interrater reliability, three evaluators graded a shared set of 15 reports and demonstrated substantial agreement with a Kendall's W of 0.71. Participants demonstrated significant improvement in overall report accuracy, clarity, and organization. In subunit analysis, there were significant improvements in reporting Le Fort, nasoseptal, and nasoorbitoethmoid fractures. No significant improvements occurred in the reporting of upper face, zygomaticomaxillary complex, or mandibular fractures. In contrast, survey analysis demonstrated significantly improved confidence in interpreting trauma involving all facial subunits. Compared with survey results, surgeon assessment of radiology reports better demonstrated areas of improvement after an educational intervention. A multidisciplinary approach to assessing educational efforts may better evaluate the practical effectiveness of educational interventions.

The Turkish version of the cognitive assessment scale for stroke patients (CASP-TR): a reliability and validity

ABSTRACT Background Stroke, a prevalent neurological condition globally, is associated with a high mortality and often affects cognitive function. The Cognitive Assessment Scale for Stroke Patients (CASP) was developed to quickly identify and measure primary cognitive deficits, focusing particularly on stroke survivors. Within the Turkish context, there is a limited availability of validated cognitive screening tests for use after stroke. Aims The study aimed to translate CASP into Turkish (CASP-TR) and validate it in stroke patients. Methods and procedures After the translation of the CASP items into culturally appropriate Turkish versions, a pilot testing of the pre-final version was administered to post-stroke patients and healthy controls by two raters. Following completion of the final version of CASP-TR, 40 stroke patients were consecutively enrolled in validation process. Three blinded raters conducted all assessments with intervals of 1- or 2 days between evaluation sessions. The accuracy of CASP-TR was assessed using sensitivity, specificity, predictive values, likelihood ratios, and area under curve. CASP-TR performance in post-stroke patients with and without cognitive dysfunction was compared using the Mini-Mental State Examination (MMSE). CASP-TR data obtained by two raters were compared based on the presence of aphasia. Outcomes and results The study assessed inter-rater agreement using intraclass correlation coefficient (ICC), yielding a score of 0.92 (95% CI 0.85–0.96) between raters. Across subtests, moderate to good agreement was observed in most domains. Correlation analyses showed a strong positive relationship (r = 0.80) between mean CASP-TR and MMSE scores. CASP-TR and MMSE scores were stratified based on the presence of aphasia. CASP-TR scores from both raters were notably lower in the aphasia group compared to the non-aphasia group (p < 0.001). Similarly, MMSE scores were significantly lower in the aphasia group (p < 0.001). The patients were also classified into cognitive dysfunction and non-cognitive dysfunction groups based on MMSE scores. No statistically significant score differences were found between examiners in the cognitive dysfunction group (p = 0.267) or non-cognitive dysfunction group (p = 0.156) (p > 0.05). CASP-TR demonstrated a sensitivity of 82.4%, specificity of 78.6%, positive predictive value of 87.5%, negative predictive value of 68.7%, a positive likelihood ratio of 3.77, a negative likelihood ratio of 0.25, and an area under curve of 0.88. Conclusions This study proposes that CASP-TR is a valuable cognitive screening tool for assessing cognitive deficits in stroke patients, and it also demonstrates sensitivity to cognitive deficits in stroke patients with aphasia. The thorough validation within the acute post-stroke Turkish population demonstrates remarkably good accuracy and very good Area Under Curve (AUC) values.

Assessment of the Intra- and Inter-Observer Reliability of Beef Cattle Mobility Scoring Performed by UK Veterinarians and Beef Farmers

Assessment of the Intra- and Inter-Observer Reliability of Beef Cattle Mobility Scoring Performed by UK Veterinarians and Beef Farmers

Standardizing Identification of Cognitive Impairment in the Acute Hospital Setting: Toward a Common Language.

Although many hospitals universally screen their patients for functional mobility and activity impairment, there is no common method to do this for cognitive impairment. Establishing an interdisciplinary and standardized process to screen for cognitive impairment is essential for early identification of impairments, optimization of patients' function, and safe discharge planning. Previously, the reliability and validity of the Activity Measure for Post-Acute Care Applied Cognitive Inpatient Short Form "6-Clicks" (AM-PAC ACISF) were measured among occupational therapy practitioners and speech-language pathologists; however, its reliability among other team members, specifically nurses, is unknown. This study reports the interprofessional interrater reliability of the AM-PAC ACISF in the acute care hospital setting. To examine the nursing-therapist interrater reliability of the AM-PAC ACISF in the acute care hospital setting. A prospective study using a convenience sample. Acute care hospital. Fifty adult patients, six nurses, five occupational therapy practitioners, and one speech-language pathologist. Interrater reliability for the AM-PAC ACISF among therapists and nurses. Interrater reliability was good among nurses and therapists, with an intraclass correlation coefficient of .88 (95% confidence interval [.79, .93]) for the AM-PAC ACISF. Findings suggest that the AM-PAC ACISF is reliable for use among interdisciplinary teams, helps to establish a common language for early identification of cognitive impairment, and can be used for guiding clinical discussions around patient cognition in the acute care setting. Plain-Language Summary: Having an interdisciplinary and standardized process to screen patients early for cognitive impairment is essential for identifying impairments, optimizing patients' function, and facilitating safe discharge planning in the acute care setting. Early identification of cognitive impairment by the interdisciplinary team can also allow for timely occupational therapy and other rehabilitation service interventions. The findings of this study suggest that the Activity Measure for Post-Acute Care Applied Cognitive Inpatient Short Form "6-Clicks" (AM-PAC ACISF) can reliably be used by interdisciplinary teams, establishes a common language for early identification of cognitive impairment, and can be used for guiding clinical discussions around patient cognition in the acute care setting.

Accuracy of the LaserSAFE technique for detecting positive surgical margins during robot-assisted radical prostatectomy: blind assessment and inter-rater agreement analysis.

Fluorescence confocal microscopy (FCM) is a new imaging modality capable of generating digital microscopic resolution scans of fresh surgical specimens, and holds potential as an alternative to frozen section (FS) analysis for intra-operative assessment of surgical margins. Previously, we described the LaserSAFE technique as an application of FCM for margin assessment in robot-assisted radical prostatectomy (RARP) using the Histolog® scanner. This study describes the accuracy and inter-rater agreement of FCM imaging compared to corresponding paraffin-embedded analysis (PA) among four blinded pathologists for the presence of positive surgical margins (PSM). RARP specimens from patients enrolled in the control arm of the NeuroSAFE PROOF study (NCT03317990) were analysed from April 2022 to February 2023. Prostate specimens were imaged using the Histolog® scanner before formalin fixation and PA. Four trained assessors, blinded to PA, reviewed and analysed FCM images of the posterolateral prostatic surface. A total of 31 prostate specimens were included in the study. PA per lateral side of the prostate identified 11 instances of positive margins. Among the four histopathologists included in our study, FCM achieved a sensitivity of 73-91 and specificity of 94-100% for the presence of PSM. Fleiss' Kappa for inter-rater agreement on PSM was 0.78 (95% confidence interval = 0.64-0.92), indicating substantial agreement. This blinded analysis of FCM versus PA among histopathologists with different experience levels demonstrated high accuracy and substantial inter-rater agreement for diagnosing PSM. This supports the role of the FCM as an alternative to FS.

Evaluation of the reliability and accuracy of MRI for the diagnosis of meniscal ramp lesions.

Meniscal ramp lesions are a special type of meniscal injury that affects the periphery of the posterior horn of the medial meniscus and/or its meniscocapsular attachments, strongly associated with anterior cruciate ligament (ACL) tears. Due to their location, these lesions can be missed arthroscopically so it is essential to diagnose them on preoperative magnetic resonance imaging (MRI). To evaluate the accuracy of MRI in detecting meniscal ramp lesions in patients with ACL tears using arthroscopy as the reference standard. Two musculoskeletal radiologists, blinded to the surgical findings, retrospectively and independently evaluated 106 knee MRI scans for the presence of meniscal ramp lesions in non-consecutive patients who underwent arthroscopic ACL reconstruction between January 2019 and July 2022 by a single surgeon at one institution. Having arthroscopy as reference, the diagnostic sensitivity and specificity as well as the positive and negative predictive values (PPV/NPV) of the MRI scans were calculated. Cohen's kappa coefficient was used to test inter-observer reliability. A P value <0.05 was considered statistically significant. In the study group of 106 patients (72 men, 34 women; mean age = 33.84 ±13.12 years), 76 had an arthroscopy-confirmed meniscal ramp lesion, while 30 did not. The sensitivity and specificity of MRI for the detection of meniscal ramp lesion were 88% and 87%, respectively. The PPV and NPV were 94% and 74%, respectively. Inter-rater reliability was excellent (k = 0915). This study demonstrates that MRI can accurately detect meniscal ramp lesions.

Reliability of a Tailored Tele-Medicine Model for Otorhinolaryngology-Head and Neck Surgery (ORL-HNS) Clinical Practice in Remote Areas of Nepal: A Pilot Study from Tertiary Referral Center.

Introduction: Despite the ever-expanding scope of telemedicine, studies regarding its application in the field of otorhinolaryngology-head and neck surgery (ORL-HNS) in remote parts of the world are lacking. This pilot study aims to bridge the research gap and to assess the efficacy of our tailored telemedicine clinic model, operating in an "asynchronous" mode, in diagnosing various ORL-HNS diseases. Methods: This is a cross-sectional study conducted at a tertiary care center of Nepal. An asynchronous telemedicine portal was used with a store and share approach to telecast the pen endoscopic camera findings to the consultant in order to formulate a diagnosis and a management plan. Interobserver reliability was assessed using SPSS version 26 based on Cohen's kappa. The association between the results of the Mimi Hearing App (v. 5.0) and pure tone audiometry were performed using chi-square test. Results: The overall diagnostic agreement in our study was substantial to almost perfect with kappa values of .843, .784, .737, and .764 for rhinology, head and neck, otology, and pediatric cases. Chronic otitis media was the predominant diagnosis across all age groups. Various head and neck swellings, symptomatic deviated nasal septum and chronic rhinosinusitis with polyposis followed closely. Discussion: The future of telemedicine in otorhinolaryngology can be gauged to be effective in reducing cost and human resource use in the limited setting of Nepal with the model allowing for remote but effective communication between the concerned patients and the expert doctor in clarifying doubts and making diagnosis, early recognition of diseases and preventing or decreasing complications of those diseases.

Evaluating pain in non-verbal critical care patients: a narrative review of the critical care pain observation tool and Its clinical applications.

Assessing pain in critically ill patients who cannot communicate verbally poses significant challenges. Traditional self-report measures are ineffective for these patients, making the need for reliable observational tools crucial. To evaluate the effectiveness, reliability, and clinical applicability of the Critical Care Pain Observation Tool (CPOT) in various intensive care unit (ICU) settings and to explore potential innovations for improving its use and integration into clinical practice. A narrative review evaluated the Critical Care Pain Observation Tool (CPOT) for non-communicative ICU patients, comparing it to the Behavioral Pain Scale (BPS) and the FLACC scale. The review assessed CPOT's effectiveness across different ICU settings, identified limitations and challenges, and explored potential enhancements such as electronic scoring, additional physiological indicators, and improved training protocols. The CPOT has been validated as an effective pain assessment tool for non-verbal ICU patients. It evaluates pain through facial expressions, body movements, muscle tension, and ventilator compliance. The CPOT shows superior sensitivity at 76.5% compared to 62.7% for the BPS and offers a more comprehensive assessment of pain indicators like muscle tension and ventilator compliance than the FLACC scale. Despite its strengths, the CPOT has limitations, including inter-rater variability and challenges in certain patient populations. Barriers to implementation include resource constraints and the need for extensive training. The Critical Care Pain Observation Tool (CPOT) is a highly effective instrument for assessing pain in non-verbal ICU patients, demonstrating superior accuracy and reliability compared to other tools like the Behavioral Pain Scale (BPS) and FLACC scale. Its detailed approach, covering facial expressions, body movements, muscle tension, and ventilator compliance, offers a detailed measure of pain. However, challenges such as inter-rater variability and limitations in specific patient populations highlight the need for ongoing refinement and research.

ChatGPT as an effective tool for quality evaluation of radiomics research.

This study aimed to evaluate the effectiveness of ChatGPT-4o in assessing the methodological quality of radiomics research using the radiomics quality score (RQS) compared to human experts. Published in European Radiology, European Radiology Experimental, and Insights into Imaging between 2023 and 2024, open-access and peer-reviewed radiomics research articles with creative commons attribution license (CC-BY) were included in this study. Pre-prints from MedRxiv were also included to evaluate potential peer-review bias. Using the RQS, each study was independently assessed twice by ChatGPT-4o and by tworadiologists with consensus. In total, 52 open-access and peer-reviewed articles were included in this study. Both ChatGPT-4o evaluation (average of two readings) and human experts had a median RQS of 14.5 (40.3% percentage score) (p > 0.05). Pairwise comparisons revealed no statistically significant difference between the readings of ChatGPT and human experts (corrected p > 0.05). The intraclass correlation coefficient for intra-rater reliability of ChatGPT-4o was 0.905 (95% CI: 0.840-0.944), and those for inter-rater reliability with human experts for each evaluation of ChatGPT-4o were 0.859 (95% CI: 0.756-0.919) and 0.914 (95% CI: 0.855-0.949), corresponding to good to excellent reliability for all. The evaluation by ChatGPT-4o took less time (2.9-3.5 min per article) compared to human experts (13.9 min per article by one reader). Item-wise reliability analysis showed ChatGPT-4o maintained consistently high reliability across almost all RQS items. ChatGPT-4o provides reliable and efficient assessments of radiomics research quality. Its evaluations closely align with those of human experts and reduce evaluation time. Question Is ChatGPT effective and reliable in evaluating radiomics research quality based on RQS? Findings ChatGPT-4o showed high reliability and efficiency, with evaluations closely matching human experts. It can considerably reduce the time required for radiomics research quality assessment. Clinical relevance ChatGPT-4o offers a quick and reliable automated alternative for evaluating the quality of radiomics research, with the potential to assess radiomics research at a large scale in the future.

Zoom to the Virtual Room: The Shift to Remote Early Childhood Observational Assessments.

The COVID-19 pandemic has led prevention researchers to increasingly use remote observational procedures given social distancing directives associated with the pandemic. However, few studies have used remote observational procedures with children and their caregivers, with scant literature describing procedures with socioeconomically, racially, and ethnically diverse families. This manuscript describes processes to pivot to remote assessment of parent and child observations in the context of a longitudinal study examining the effects of a postpartum depression preventive intervention on responsive parenting and child self-regulation. We conducted remote assessments across three timepoints-42, 48, and 54 months-with 133 low-income and racially and ethnically diverse parent-child dyads. Details are provided on remote observation preparation and setup, as well as adaptation of observational assessments. Lessons learned are shared on the use of technology, scheduling considerations, parent's role as facilitator, maintaining child engagement, and cost considerations. We demonstrated excellent inter-rater reliability between independent coders on all assessments, suggesting the quality of remote assessments was conducive for analysis. Surveys with families completing a remote assessment found that most felt it was easy to participate in remote assessments and their child had a positive experience. Most parents preferred virtual visits if given an option for future assessments. Much prevention research utilizes observational measures that are less subject to bias. Our study demonstrated that research teams can effectively and reliably pivot to remote assessment with racially and ethnically diverse, low socioeconomic families, thereby providing guidance to other prevention researchers considering similar remote assessments with diverse samples.

Multicenter validation of automated detection of paramagnetic rim lesions on brain MRI in multiple sclerosis.

Paramagnetic rim lesions (PRLs) are an MRI biomarker of chronic inflammation in people with multiple sclerosis (MS). PRLs may aid in the diagnosis and prognosis of MS. However, manual identification of PRLs is time-consuming and prone to poor interrater reliability. To address these challenges, the Automated Paramagnetic Rim Lesion (APRL) algorithm was developed to automate PRL detection. The primary objective of this study is to evaluate the accuracy of APRL for detecting PRLs in a multicenter setting. We applied APRL to a multicenter dataset, which included 3-Tesla MRI acquired in 92 participants (43 with MS, 14 with clinically isolated syndrome [CIS]/radiologically isolated syndrome [RIS], 35 without RIS/CIS/MS). Subsequently, we assessed APRL's performance by comparing its results with manual PRL assessments carried out by a team of trained raters. Among the 92 participants, expert raters identified 5637 white matter lesions and 148 PRLs. The automated segmentation method successfully captured 115 (78%) of the manually identified PRLs. Within these 115 identified lesions, APRL differentiated between manually identified PRLs and non-PRLs with an area under the curve (AUC) of.73 (95% confidence interval [CI]: [.68,.78]). At the subject level, the count of APRL-identified PRLs predicted MS diagnosis with an AUC of.69 (95% CI: [.57,.81]). Our study demonstrated APRL's capability to differentiate between PRLs and lesions without paramagnetic rims in a multicenter study. Automated identification of PRLs offers greater efficiency over manual identification and could facilitate large-scale assessments of PRLs in clinical trials.

VASARI-AUTO: PERFORMANT, EQUITABLE, EffiCIENT, ECONOMIC, AND SURVIVAL PREDICTIVE FEATURISATION OF GLIOMA MRI

Abstract AIMS The VASARI (Visually AcceSAble Rembrandt Images) MRI feature set is a quantitative system designed to standardize glioma imaging descriptions. Though effective, deriving VASARI is time-consuming and therefore seldom used in clinical practice. This is however a problem that performant machine-learning software could plausibly automate. METHOD In 100 patients, two consultant neuroradiologists independently quantified VASARI features. In parallel, we developed VASARI-auto, an automated VASARI-labelling software applied to both open-source hand-segmented lesion masks and our openly available tumour segmentation model. We quantified: 1) agreement across neuroradiologists and VASARI-auto; 2) calibration of patient equity; 3) an economic workforce analysis; and 4) fidelity in predicting patient overall survival. Neuroradiologists were blinded to VASARI-auto software development and evaluations, and software developers were blinded to neuroradiologist labelling. RESULTS Segmentation and VASARI-auto were equally performant regardless of age or sex (mean segmentation Dice coefficient 0.947). A modest inter-rater variability between neuroradiologists (mean Cohen’s Kappa 0.49) was comparable between neuroradiologists and VASARI-auto (mean Cohen’s Kappa 0.41), with higher agreement across VASARI-auto methods (mean Cohen’s Kappa 0.94). The time taken for neuroradiologists to derive VASARI features was substantially higher than VASARI-auto (mean time per case 317 vs. 3 seconds, p&amp;lt;0.0001). A hospital workforce analysis of our centre forecast that three years of VASARI glioma featurisation would demand 771 consultant neuroradiologist workforce hours (£40,789) or 8.6 hours of computing time (£3.76 of power) with VASARI-auto. The best-performing survival model utilised VASARI-auto features (R2 0.25), opposed to those derived by consultant neuroradiologists (R2 0.21). CONCLUSION VASARI-auto is a highly efficient automated labelling system for VASARI featurisation with equitable performance regardless of patient age or sex, a favourable economic profile if used as a decision support tool, and with non-inferior fidelity in downstream patient survival prediction. Future work should integrate such tools to enhance patient care.

Reliability and Cross-Cultural Validation of the Turkish Version of the Visual Function Classification System (VFCS) for Children with Cerebral Palsy

ABSTRACT Purpose This correlational study aimed to determine the reliability and validity of the Turkish version of the Visual Function Classification System (VFCS) for children with CP. Method Two physiotherapists and caregivers (n = 56) applied the VFCS (n = 120) two times at a 15-day interval. Physiotherapists’ classifications were compared for interrater reliability. Same physiotherapists and caregivers’ classifications’ were compared for intrarater reliability. Expanded & Revised Gross Motor Function Classification System (GMFCS E&R), Manual Ability Classification System (MACS), and Communication Function Classification System (CFCS) were used for assessing validity of the culturally adapted VFCS. Results The interrater reliability between physiotherapists was excellent (ICC = 0.974, ICC = 0.988), while interrater reliability between the first physiotherapist and caregivers was good (ICC = 0.866, ICC = 0.893). The intrarater reliability was excellent (ICC = 0.971, ICC = 0.996, ICC = 0.984). A correlation was detected between the VFCS and the GMFCS E&R, MACS, and CFCS levels (r = 0.415, p < .001, r = 0.450, p < .001, r = 0.491, p < .001). Conclusion The findings of this study demonstrate the utility, compatibility and reliability of the Turkish version of the VFCS. Trial registration number NCT05102955

Reliability assessment of the ‘field audit for children’s active transport routes to school’ (FACTS) tool

BackgroundChildren’s active travel to school is associated with physical activity and thus health. Safe Routes to School (SR2S) programs identify ‘safe routes’ to promote children’s active travel to school. No field audit tool exists specifically to assess the microscale built environment of these ‘safe routes’ within Australia. This study describes the reliability assessment of the Field Audit for Children’s Active Transport to School (FACTS) tool.MethodsThe FACTS tool was developed using a multi-step process, including a literature search, expert opinion, and pilot testing. For the reliability assessment, two trained auditors collected data at eight schools over three weeks in April 2021. For inter-rater reliability, auditors conducted audits on the ‘safe routes’ for the same six schools. For intra-rater reliability, auditors conducted repeat measures on the ‘safe routes’ for four schools each (eight schools total), including three different schools each from the six used for inter-rater reliability and one additional school. Item-by-item reliability was assessed using Cohen’s Kappa, Cohen’s Weighted Kappa, and percentage agreement. The reliability of calculated domain scores was assessed using intraclass correlation coefficients.ResultsFor inter-rater reliability, 31 of the 45 (68.9%) items had moderate to almost perfect agreement, seven items (15.6%) had below moderate agreement, and a Kappa statistic could not be calculated for seven items (15.6%) due to constant values. For intra-rater reliability, 37 of the 45 (82.2%) items had moderate to almost perfect agreement, two items (4.4%) had below moderate agreement, and a Kappa statistic could not be calculated for six items (13.3%) due to constant values. For inter- and intra-rater reliability of the segment domain scores, three of the four domains had substantial to almost perfect agreement. For inter- and intra-rater reliability of the crossing domain scores, all four domains had moderate to almost perfect agreement. For inter- and intra-rater reliability of the segment, crossing, route, and school scores, all had substantial to perfect agreement.ConclusionsThe FACTS tool can reliably characterise the microscale built environment of promoted ‘safe routes’ for their use within SR2S programs, and should be considered for use in future SR2S programs within the suburban Australian context.

Comparing Scoring Consistency of Large Language Models with Faculty for Formative Assessments in Medical Education.

The Liaison Committee on Medical Education requires that medical students receive individualized feedback on their self-directed learning skills. Pre-clinical students are asked to complete multiple spaced critical appraisal assignments. However, the individual feedback requires significant faculty time. As large language models (LLMs) can score and generate feedback, we explored their use in grading formative assessments through validity and feasibility lenses. To explore the consistency and feasibility of using an LLM to assess and provide feedback for formative assessments in undergraduate medical education. This was a cross-sectional study of pre-clinical students' critical appraisal assignments at University of Illinois College of Medicine (UICOM) during the 2022-2023 academic year. An initial sample of ten assignments was used to develop a prompt. For each student entry, the de-identified assignment and prompt were provided to ChatGPT 3.5, and its scoring was compared to the existing faculty grade. Differences in scoring of individual items between ChatGPT and faculty were assessed. Scoring consistency using inter-rater reliability (IRR) was calculated as percent exact agreement. Chi-squared test was used to determine if there were significant differences in scores. Psychometric characteristics including internal-consistency reliability, area under precision-recall curve (AUCPR), and cost were studied. In this cross-sectional study, 111 pre-clinical students' faculty graded assignments were compared with that of ChatGPT and the scoring of individual items was comparable. The overall agreement between ChatGPT and faculty was 67% (OR = 2.53, P < 0.001); mean AUCPR was 0.69 (range 0.61-0.76). Internal-consistency reliability of ChatGPT was 0.64 and its use resulted in a fivefold reduction in faculty time, and potential savings of 150 faculty hours. This study of psychometric characteristics of ChatGPT demonstrates the potential role for LLMs to assist faculty in assessing and providing feedback for formative assignments.

EthoWatcher OS: improving the reproducibility and quality of categorical and morphologic/kinematic data from behavioral recordings in laboratory animals.

Behavioral recordings annotated by human observers (HOs) from video recordings are a fundamental component of preclinical animal behavioral models of neurobiological diseases. These models are often criticized for their vulnerability to reproducibility issues. Here, we present the EthoWatcher-Open Source (EW-OS), with tools and procedures for the use of blind-to-condition categorical transcriptions that are simultaneous with tracking, for the assessment of HOs intra- and interobserver reliability during training and data collection, for producing video clips of samples of behavioral categories that are useful for observer training. The use of these tools can inform and optimize the performance of observers, thus favoring the reproducibility of the data obtained. Categorical and machine vision-derived outputs are presented in an open data format for increased interoperability with other applications, where behavioral categories are associated frame-by-frame with tracking, morphological and kinematic attributes of an animal's image. The center of mass (X and Y pixel coordinates), the animal's area in square millimeters, the length and width in millimeters, and the angle in degrees were recorded. It also assesses the variation in each morphological descriptor to produce kinematic descriptors. While the initial measurements are in pixels, they are later converted to millimeters using the scale calibrated by the user via the graphical user interfaces. This process enables the creation of databases suitable for machine learning processing and behavioral pharmacology studies. EW-OS is constructed for continued collaborative development, available through an open-source platform, to support initiatives toward the adoption of good scientific practices in behavioral analysis, including tools for evaluating the quality of the data that can alleviate problems associated with low reproducibility in the behavioral sciences.

Measuring clinical outcomes in adult ADHD clinics: psychometrics of a new scale, the adult ADHD Clinical Outcome Scale.

Adult attention-deficit hyperactivity disorder (ADHD) clinics are in their infancy in Ireland and internationally. There is an urgent need for clinical evaluation of these services. Until now, clinical outcomes have relied mainly on functional scales and/or quality of life. However, adult ADHD is a longstanding disorder with many comorbidities. Although medication for ADHD symptoms can have immediate effects, co-occurring problems may take considerably longer to remediate. To present the psychometrics of a short outcome measure of key clinical areas including symptoms. The ADHD Clinical Outcome Scale (ACOS), developed by the authors, is a clinician-rated scale and was administered in consecutive adults attending an ADHD clinic. A modified version was completed by the participant. A second clinician independently administered the scale in a subsample. ACOS consists of 15 items rated on a Likert scale. Two self-report scales, the Adult ADHD Quality of Life Questionnaire (AAQoL) and Weiss Functional Impairment Rating Scale (WFIRS), were also administered. The mean age of 148 participants was 30.1 years (s.d. = 9.71), and 81 were female (54.7%). The correlation for interrater reliability was r = 0.868, and that between the participant and clinician versions was r = 0.663. The intraclass correlation coefficient for the internal consistency was 0.829, and the correlations for concurrent validity with total AAQoL and WFIRS scores were r = -0.573 and r = 0.477, respectively. Factor analysis revealed four factors: (a) attentional/organisational problems; (b) hyperactivity/impulsivity; (c) comorbidities; and (d) alcohol/drug use, self-harm and tension in relationships. The psychometrics of the ACOS are promising, and the inclusion of typically co-occurring clinical domains makes it suitable for use as a clinician-rated outcome measure in every contact with patients attending adult ADHD clinics.

Perceptual and conceptual novelty independently guide infant looking behaviour: a systematic review and meta-analysis.

Human infants are born with their eyes open and an otherwise limited motor repertoire; thus, studies measuring infant looking are commonly used to investigate the developmental origins of perception and cognition. However, scholars have long expressed concerns about the reliability and interpretation of looking behaviours. We evaluated these concerns using a pre-registered ( https://osf.io/jghc3 ), systematic meta-analysis of 76 published and unpublished studies of infants' early physical and psychological reasoning (total n = 1,899; 3- to 12-month-old infants; database search and call for unpublished studies conducted July to August 2022). We studied two effects in the same datasets: looking towards expected versus unexpected events (violation of expectation (VOE)) and looking towards visually familiar versus visually novel events (perceptual novelty (PN)). Most studies implemented methods to minimize the risk of bias (for example, ensuring that experimenters were naive to the conditions and reporting inter-rater reliability). There was mixed evidence about publication bias for the VOE effect. Most centrally to our research aims, we found that these two effects varied systematically-with roughly equal effect sizes (VOE, standardized mean difference 0.290 and 95% confidence interval (0.208, 0.372); PN, standardized mean difference 0.239 and 95% confidence interval (0.109, 0.369))-but independently, based on different predictors. Age predicted infants' looking responses to unexpected events, but not visually novel events. Habituation predicted infants' looking responses to visually novel events, but not unexpected events. From these findings, we suggest that conceptual and perceptual novelty independently influence infants' looking behaviour.

Image quality of photon-counting detector CT for visualization of maxillofacial anatomy in comparison with energy-integrating detector CT and intraoperative C-arm CBCT

BackgroundAccurate diagnostic imaging is crucial for managing facial fractures, which are a common global occurrence. This study aimed to compare the image quality of Photon Counting Detector CT (PCD-CT) with state-of-the-art Energy Integrating Detector CT (EID-CT) and intraoperative C-arm CBCT (CBCT) in visualizing maxillofacial anatomy using a cadaveric sheep head model. MethodsThree fresh sheep heads were used, with surgical interventions simulating metal implants in two of them. The specimens were imaged using PCD-CT, EID-CT, and CBCT, following which quantitative assessments of signal-to-noise ratio, sharpness, and artifacts were conducted. A visual grading study was performed by six observers, using criteria focusing on the mandible, orbit, and soft tissues. Statistical analyses included Friedman tests for comparing modalities and Kendall’s W and Gwet’s AC1 for assessing inter- and intrarater agreement. ResultsPCD-CT demonstrated a significantly higher signal-to-noise ratio (p = 0.03) and bone sharpness (p < 0.001) compared to CBCT. In visual grading, PCD-CT outperformed CBCT, but not EID-CT, particularly in delineating mandibular and orbital structures. EID-CT and PCD-CT showed slightly more severe hypodense artifacts (p = 0.01) but were comparable in streak artifact presentation. The interrater and intrarater agreements indicated consistent evaluations across and within observers. ConclusionPCD-CT exhibits superior image quality over CBCT in key parameters essential for maxillofacial imaging, while no apparent improvement was shown compared to state-of-the-art EID-CT. PCD-CT offers enhanced visualization of critical anatomical structures, suggesting its potential as a preferred modality in managing maxillofacial trauma. The findings in this study align with limited existing research on PCD-CT, underscoring its promise for advanced diagnostic imaging in maxillofacial applications.

The Effect of Gossiping Group Activities Toward Fluency in Speaking at the 10th Grade Of MAN 3 Agam

This research was motivated by the findings of the researcher during observations and interviews at the 10th grade of MAN 3 Agam. Researcher found that students often used pause fillers such as "hmm" and "eee", and had low vocabulary mastery that hindered fluency in speaking. They also had difficulty expressing ideas and opinions, so the speed and tempo of speaking were affected. The study aimed to determine the effect of gossiping group activities on students' speaking fluency. The study used a type of experimental research, namely a quasi-experiment with a nonequivalent control group design. The sample in this study was selected using a random technique and the selected class was XE.2 class as the control class and XE.4 class as the experimental class. Instruments in the form of oral speaking tests that had been validated with content validity and assessed with inter-rater reliability. Data analysis included prerequisite tests for normality and homogeneity, and utilized Paired sample t-Tests and Independent sample t-Tests to test hypotheses. From the results of data analysis, it was concluded that gossiping group activities had a significant effect on improving students' speaking fluency.