Substantial Inter-rater Reliability Research Articles

Capability of ChatGPT to support the screening process of scoping reviews: A feasibility study

Abstract Background The time-consuming screening phase in health-related evidence syntheses is increasingly supported by artificial intelligence (AI). However, scoping reviews have not benefited as much as systematic reviews from such AI tools as they utilize conceptual rather than keyword-specific search strategies to address broad research questions. Context-understanding chatbots based on large language models could potentially enhance the efficiency of scoping review screenings. This study evaluates the performance of ChatGPT against an open-access AI tool used for abstract screening in systematic reviews and the costs involved. Methods Leveraging data from a prior scoping review, we compared the performance of ChatGPT 4.0 and 3.5 against Rayyan, using human researchers’ decisions as a benchmark. A random set of 50 included and 50 excluded abstracts was used to train Rayyan’s algorithm and to develop ChatGPT’s prompt. ChatGPT 4.0’s evaluation was repeated after 5-7 days to assess response consistency. We computed performance metrics including sensitivity, specificity, and accuracy. Results ChatGPT 4.0 and 3.5 achieved 68% accuracy, 11% precision, 99% negative predictive value, and 67% specificity. Sensitivity was high at 88-89% for ChatGPT 4.0 and 84% for ChatGPT 3.5. ChatGPT 4.0 showed a substantial interrater reliability between the two evaluations and moderate reliability compared to ChatGPT 3.5. The cost of deployment varied, with Rayyan being free, ChatGPT 3.5 costing $9.06 and ChatGPT 4.0 $505.72. Conclusions Given the exponential increase in publications, effective mechanisms to support the screening phase of scoping reviews are needed. Our feasibility study using ChatGPT to decide on abstracts’ inclusion or exclusion achieved good performance metrics. Given further positive evaluation, such chatbots might be incorporated in the standard screening process, possibly replacing a second screener, saving time and costs, and accelerating evidence synthesis. Key messages • ChatGPT performs well in supporting screening for scoping reviews, outperforming a traditional AI tool at reasonable costs. • Employing chatbots like ChatGPT could potentially cut costs and time in scoping reviews.

Reliability of an Expanded Non-alcoholic Steatohepatitis Grading and Staging System for Assessment of Disease Activity and Fibrosis.

The nonalcoholic steatohepatitis clinical research network (NASH-CRN) system is commonly used for histologic assessment of disease activity and fibrosis in NASH. Despite this, the system does not fully capture the range of disease activity and fibrosis. As such, an expanded NAS (E-NAS) grading and staging system with a calculated E-NAS index was developed by our group. In this follow up study, we aim to revalidate the E-NAS system and compare its reliability to existing systems. Hematoxylin and eosin and trichrome stained sections from 40 liver biopsies were reviewed digitally by four hepatopathologists and assessed using the NASH-CRN and E-NAS systems as well as a modified Ishak fibrosis stage. The pathologist's gestalt impression of disease activity and fibrosis was scored on a Visual Analogue Scale (VAS), which ranged from 0 (no activity/fibrosis) to 100 (the worst activity/fibrosis ever seen). Inter-rater reliability was assessed, and Spearman correlation coefficients were calculated. The E-NAS index had higher inter-rater agreement versus the NAS score (ICC 0.70 vs 0.61). The inter-rater agreement for ballooning in the E-NAS system was also higher at 0.67 compared to the NAS (ICC 0.60). ICCs for fibrosis were comparable between all the systems assessed (0.78 to 0.88). Finally, the calculated E-NAS index was higher with increasing stage of fibrosis compared to the NAS suggesting that it associates better with fibrosis. In summary, the E-NAS system demonstrates substantial inter-rater reliability as well as improved correlation with disease activity VAS and fibrosis compared to the NAS score.

The Development and Validation of the Emotionally Focused Therapy Skills Rubric

This article outlines the initial psychometric evidence for the emotionally focused therapy skills rubric (EFTSR). The EFTSR is designed to assess clinician's emotionally focused therapy (EFT) skill proficiency, identify how EFT skills are used within the basic movements of EFT, and evaluate a clinician's ability to use the EFT attachment injury resolution model . The EFTSR content validity was confirmed with two overall content validity indexes of .86 and .87. There was substantial item-level internal consistency reliability, with α scores ranging from .70 to .99 for both videos. Item-total correlations were higher than .20 for most items demonstrating substantial interrater reliability. Thus, results indicate initial support for using the EFTSR as theoretically based instrument designed to assess counselors’ ability to execute the skills of the EFT model.

The Echocardiographic Diagnosis of Rheumatic Heart Disease: A Review of the Performance of the World Heart Federation Criteria 2012-2023.

The World Heart Federation (WHF) published the first evidence-based guidelines on the echocardiographic diagnosis of rheumatic heart disease (RHD) in 2012. These guidelines have since been applied internationally in research and clinical practice. Substantial research has assessed the utility of the 2012 WHF criteria, including its applicability in low-resource settings. This article summarises the evidence regarding the performance of the guidelines. A scoping review assessing the performance of the guidelines was performed. Cochrane, Embase, Medline, PubMed Lilacs, Sielo, and Portal BVS databases were searched for studies on the performance of the guidelines between January 2012-March 2023, and 4047 manuscripts met the search criteria, of which 34 were included. This included papers assessing the specificity, inter-rater reliability, application using hand-carried ultrasound, and modification of the criteria for simplicity. The review followed the PRISMA Extension for Scoping Reviews guideline. The WHF 2012 criteria were 100% specific for definite RHD when applied in low-prevalence populations. The criteria demonstrated substantial and moderate inter-rater reliability for detecting definite and borderline RHD, respectively. The inter-rater reliability for morphological features was lower than for valvular regurgitation. When applied to hand-carried ultrasound performed by an expert, modified versions of the criteria demonstrated a sensitivity and specificity range of 79-90% and 87-93% respectively for detecting any RHD, performing best for definite RHD. The sensitivity and the specificity were reduced when performed in task-sharing but remains moderately accurate. The WHF 2012 criteria provide clear guidance for the echocardiographic diagnosis of RHD that is reproducible and applicable to a range of echocardiographic technology. Furthermore, the criteria are highly specific and particularly accurate for detecting definite RHD. There are limitations in applying all aspects of the criteria in specific settings, including task-sharing. This summary of evidence can inform the updated version of the WHF guidelines to ensure improved applicability in all RHD endemic regions.

Reliability and Responsiveness of Clinical and Endoscopic Outcome Measures in Crohn's Disease.

Regulatory guidance for Crohn's disease trials recommends coprimary efficacy end points that evaluate both symptoms and mucosal inflammation. We aimed to characterize the operating properties of commonly used disease activity assessments alone and in combination. Endoscopic and clinical data were available for 129 participants from the Study of Biologic and Immunomodulator Naïve Patients in Crohn's Disease trial. Readers scored the Simple Endoscopic Score for Crohn's Disease and the Crohn's Disease Endoscopic Index of Severity using standardized conventions. Index reliability was determined using intraclass correlation coefficients. Index responsiveness was assessed using standardized effect sizes based upon treatment assignment. Outcomes were evaluated for optimal sensitivity to treatment effect. Substantial inter-rater reliability was observed when the Simple Endoscopic Score for Crohn's Disease and Crohn's Disease Endoscopic Index of Severity were used as continuous measures (intraclass correlation coefficient, 0.64; 95% confidence interval [CI], 0.50-0.73; and 0.62 95% CI, 0.36-0.77) compared with moderate reliability when dichotomized (0.46; 95% CI, 0.26-0.65; and 0.51; 95% CI, 0.00-0.78). The Simple Endoscopic Score for Crohn's Disease, Crohn's Disease Endoscopic Index of Severity, patient-reported outcome-2, and Crohn's Disease Activity Index were similarly responsive (standardized effect size, 0.43, 95% CI, 0.05-0.81; 0.38, 95% CI, 0.0-0.76; 0.53, 95% CI, 0.15-0.91). A composite outcome of Crohn's Disease Activity Index score <150 and Crohn's Disease Endoscopic Index of Severity score <6 was most sensitive to treatment effect (28.9%; 95% CI, 11.0%-46.8%; P = .003). Endoscopic indices were more reliable as continuous measures. Composite outcomes including endoscopy improved sensitivity to treatment effect.

DeepD3, an open framework for automated quantification of dendritic spines.

Dendritic spines are the seat of most excitatory synapses in the brain, and a cellular structure considered central to learning, memory, and activity-dependent plasticity. The quantification of dendritic spines from light microscopy data is usually performed by humans in a painstaking and error-prone process. We found that human-to-human variability is substantial (inter-rater reliability 82.2±6.4%), raising concerns about the reproducibility of experiments and the validity of using human-annotated 'ground truth' as an evaluation method for computational approaches of spine identification. To address this, we present DeepD3, an open deep learning-based framework to robustly quantify dendritic spines in microscopy data in a fully automated fashion. DeepD3's neural networks have been trained on data from different sources and experimental conditions, annotated and segmented by multiple experts and they offer precise quantification of dendrites and dendritic spines. Importantly, these networks were validated in a number of datasets on varying acquisition modalities, species, anatomical locations and fluorescent indicators. The entire DeepD3 open framework, including the fully segmented training data, a benchmark that multiple experts have annotated, and the DeepD3 model zoo is fully available, addressing the lack of openly available datasets of dendritic spines while offering a ready-to-use, flexible, transparent, and reproducible spine quantification method.

Adapting the Client Priority Rating Scale to better fit the sexual health counselling setting: a quality improvement study.

When demand for counselling in community-based clinics exceeds capacity, waiting lists are typically formed. Determining allocation priority solely on wait time can overlook client risk factors that can elevate priority. We undertook to rigorously adapt the only existing validated counselling triage tool, to better fit the sexual health setting. Sexual health counsellors were surveyed about aspects of client presentations that flagged increased priority. The revised Client Priority Rating Scale (CPRS-R) was created through systematic analysis and decision making, informed by survey results and literature review. Four expert sexual health counsellors independently rated the priority of 14 hypothetical clinical vignettes using the CPRS and CPRS-R. Criterion (concurrent), content and face validity are evidenced in the revised scale. Average interrater agreement was higher on the CPRS-R (28%) than the CPRS (11%); however, this difference was marginal (P =0.06). According to Gwet's Agreement Coefficient (AC) and Krippendorff's Alpha, both the CPRS and the CPRS-R demonstrate comparable interrater reliability, substantial and moderate, respectively. Kendall's W indicates the CPRS yielded higher reliability. However, the difference is not substantial. The CPRS-R is a triage tool designed for the sexual health counselling setting. This tool has demonstrated criterion, content and face validity, as well as moderate to substantial inter-rater reliability. It can be used in sexual health settings to inform assessments about client priority, along with clinical judgement and peer consultation.

Reliability and Validity of Visual Estimation in Determining Thorax Rotation Mobility using the Quadruped Lumbar-Locked Position.

Thoracic rotation mobility is crucial for athletes in rotational sports such as baseball, golf, and swimming to maintain the proper biomechanics associated with the sport. Accurate differentiation between normal mobility and active and passive physiological deficits in the thoracic region is critical for identifying the need for intervention to the thorax. To establish the reliability and discriminant validity of visual estimation of thorax rotation range of motion across clinicians of differing experience levels in determining normal mobility and active or passive physiological deficits when utilizing the quadruped lumbar-locked position. Cross-sectional. Thirty-eight subjects (21 female, 17 male) with a mean age of 27 years ± 6.67 were assessed with the quadruped lumbar-locked thorax rotation test by three examiners with various clinical experience in real-time and again one week later. Bilateral active and passive lumbar-locked thorax rotation mobility was assessed by all raters and categorized as "Unrestricted" (≥50°) or "Restricted" (<50°) while a research assistant simultaneously measured the motion with a digital inclinometer. All raters were blinded to the results. All results were analyzed for intra-rater reliability and agreement. Test-retest intra-rater reliability ranged from 0.55-0.72 and percent absolute agreement ranged from 0.82-0.89. Inter-rater reliability ranged from 0.45-0.59 while percent absolute agreement between raters ranged from 0.74-0.84. There was a significant difference in range of motion between "Unrestricted" and "Restricted" categories for both active (Unrestricted=54.6-58.9; Restricted=40.4-44.4; p<0.001) and passive motion (Unrestricted=61.3-63.5; Restricted=39.2-39.7; p<0.001). The only interaction effect was for passive left rotation [Rater A Restricted x ® =34.3(30.4-38.2); Rater C Restricted (x ) ®=43.8(41.3-46.4); p=.000]. The quadruped lumbar-locked thorax rotation test demonstrates moderate to substantial test-retest intra-rater and inter-rater reliability regardless of clinical experience. The quadruped lumbar-locked thorax rotation test can accurately discriminate between individuals with active and passive physiological deficits regardless of rater experience using visual estimation. 3b.

Inter-rater reliability of risk of bias tools for non-randomized studies

PurposeThere is limited knowledge on the reliability of risk of bias (ROB) tools for assessing internal validity in systematic reviews of exposure and frequency studies. We aimed to identify and then compare the inter-rater reliability (IRR) of six commonly used tools for frequency (Loney scale, Gyorkos checklist, American Academy of Neurology [AAN] tool) and exposure (Newcastle–Ottawa scale, SIGN50 checklist, AAN tool) studies.MethodsSix raters independently assessed the ROB of 30 frequency and 30 exposure studies using the three respective ROB tools. Articles were rated as low, intermediate, or high ROB. We calculated an intraclass correlation coefficient (ICC) for each tool and category of ROB tool. We compared the IRR between ROB tools and tool type by inspection of overlapping ICC 95% CIs and by comparing their coefficients after transformation to Fisher’s Z values. We assessed the criterion validity of the AAN ROB tools by calculating an ICC for each rater in comparison with the original ratings from the AAN.ResultsAll individual ROB tools had an IRR in the substantial range or higher (ICC point estimates between 0.61 and 0.80). The IRR was almost perfect (ICC point estimate > 0.80) for the AAN frequency tool and the SIGN50 checklist. All tools were comparable in IRR, except for the AAN frequency tool which had a significantly higher ICC than the Gyorkos checklist (p = 0.021) and trended towards a higher ICC when compared to the Loney scale (p = 0.085). When examined by category of ROB tool, scales, and checklists had a substantial IRR, whereas the AAN tools had an almost perfect IRR. For the criterion validity of the AAN ROB tools, the average agreement between our raters and the original AAN ratings was moderate.ConclusionAll tools had substantial IRRs except for the AAN frequency tool and the SIGN50 checklist, which both had an almost perfect IRR. The AAN ROB tools were the only category of ROB tools to demonstrate an almost perfect IRR. This category of ROB tools had fewer and simpler criteria. Overall, parsimonious tools with clear instructions, such as those from the AAN, may provide more reliable ROB assessments.

Toward Efficient, Sustainable, and Scalable Methods of Treatment Characterization: An Investigation of Coding Clinical Practice from Chart Notes.

Chart notes provide a low-cost data source that could help characterize what occurs in treatment with sufficient precision to improve management of care. This study assessed the interrater reliability of treatment content coded from chart notes and evaluated its concordance with content coded from transcribed treatment sessions. Fifty randomly selected and digitally recorded treatment events were transcribed and coded for practice content. Independent coders then applied the same code system to chart notes for these same treatment events. We measured reliability and concordance of practice occurrence and extensiveness at two levels of specificity: practices (full procedures) and steps (subcomponents of those procedures). For chart notes, practices had moderate interrater reliability (M k = 0.50, MICC = 0.56) and steps had moderate (M ICC = 0.74) to substantial interrater reliability (M k = 0.78). On average, 2.54 practices and 5.64 steps were coded per chart note and 4.53 practices and 13.10 steps per transcript. Across sources, ratings for 64% of practices and 41% of steps correlated significantly, with those with significant correlations generally demonstrating moderate concordance (practice M r = 0.48; step M r = 0.47). Forty one percent of practices and 34% of steps from transcripts were also identified in the corresponding chart notes. Chart notes provide an accessible data source for evaluating treatment content, with different levels of specificity posing tradeoffs for validity and reliability, which in turn may have implications for chart note interfaces, training, and new metrics to support accurate, reliable, and efficient measurement of clinical practice.

Empowering Medical Students: Harnessing Artificial Intelligence for Precision Point-of-Care Echocardiography Assessment of Left Ventricular Ejection Fraction

Introduction. Point-of-care ultrasound (POCUS) use is now universal among nonexperts. Artificial intelligence (AI) is currently employed by nonexperts in various imaging modalities to assist in diagnosis and decision making. Aim. To evaluate the diagnostic accuracy of POCUS, operated by medical students with the assistance of an AI-based tool for assessing the left ventricular ejection fraction (LVEF) of patients admitted to a cardiology department. Methods. Eight students underwent a 6-hour didactic and hands-on training session. Participants used a hand-held ultrasound device (HUD) equipped with an AI-based tool for the automatic evaluation of LVEF. The clips were assessed for LVEF by three methods: visually by the students, by students + the AI-based tool, and by the cardiologists. All LVEF measurements were compared to formal echocardiography completed within 24 hours and were evaluated for LVEF using the Simpson method and eyeballing assessment by expert echocardiographers. Results. The study included 88 patients (aged 58.3 ± 16.3 years). The AI-based tool measurement was unsuccessful in 6 cases. Comparing LVEF reported by students’ visual evaluation and students + AI vs. cardiologists revealed a correlation of 0.51 and 0.83, respectively. Comparing these three evaluation methods with the echocardiographers revealed a moderate/substantial agreement for the students + AI and cardiologists but only a fair agreement for the students’ visual evaluation. Conclusion. Medical students’ utilization of an AI-based tool with a HUD for LVEF assessment achieved a level of accuracy similar to that of cardiologists. Furthermore, the use of AI by the students achieved moderate to substantial inter-rater reliability with expert echocardiographers’ evaluation.

Reliability and Validity Analysis of the Distal Radioulnar Joint Ballottement Test

Triangular fibrocartilage complex injuries can cause distal radioulnar joint (DRUJ) instability, which can be evaluated clinically with the DRUJ ballottement test. However, the reliability and validity of the test are unclear. This study aimed to analyze the reliability and validity of the test using a tracking device on healthy participants and patients with triangular fibrocartilage complex injuries. In this cross-sectional study, three orthopedic hand surgeons performed the DRUJ ballottement test using a technique of holding the carpal bones to the radius on 25 healthy participants (50 hands; 10 men and 15 women; mean age, 33 years; range, 20-51 years) and eight patients with triangular fibrocartilage complex injuries (16 hands; six men and two women; mean age, 43 years; range, 27-59 years). We used a three-dimensional electromagnetic tracking device to quantify the movement of the DRUJ and verify the reliability and validity of the test. The intrarater and interrater intraclass correlation coefficients for DRUJ movement were 0.77 and 0.61, respectively, and the kappa coefficient for grading of DRUJ instability was 0.79. The correlation coefficient between DRUJ movement measured using the tracking device and instability judged clinically was 0.77. A comparison of healthy participants and the patients showed significantly greater DRUJ movement in the patients. The test showed substantial intrarater and interrater reliability for assessing DRUJ movement and instability. The significant positive correlation between DRUJ movement and instability indicates the concurrent validity of the test. Moreover, the test showed discriminative validity in identifying mild or moderate DRUJ instability. The DRUJ ballottement test using the holding technique has a relatively high diagnostic accuracy and can be used to assess DRUJ instability.

Reliability of the global cortical atrophy visual rating scale applied to computed tomography versus magnetic resonance imaging scans in acute stroke

IntroductionResearch using magnetic resonance imaging (MRI) suggests regional cerebral atrophy measures (e.g., frontal lobe, temporal lobe) may predict post-stroke outcomes. Clinical CT scans have excellent potential for use in research but it is unclear whether regional atrophy measures from CT are reliable compared to MRI reference standards.MethodsWe used the Global Cortical Atrophy (GCA) scale to investigate reliability of atrophy measures on CT versus MRI scans from stroke patients originally recruited to the Oxford Cognitive Screening programme. Two raters provided standardised visual ratings at two timepoints. Weighted Kappa statistics assessed the reliability of regional atrophy scores. Spearman’s correlation and a two-way repeated measures ANOVA assessed the reliability of the total score.ResultsOn clinically acquired neuroimaging from 98 stroke patients (mean/SD age = 70.97/11.99, 42 female, 84 ischaemic stroke), regional GCA scores on CT versus MRI showed fair to almost perfect intra-rater agreement (κ = .50–.87), substantial to almost perfect intra-rater agreement on CT (κ = .67–.88), and moderate to almost perfect intra-rater reliability on MRI (κ = .50–.89). Regional GCA scores showed mostly moderate to substantial inter-rater reliability on both CT and MRI (κ = .43–.69), except the temporal horns and parieto-occipital region. There was a strong correlation between total GCA scores on CT and MRI (r (96) = .87–.88, p < .001).ConclusionsThese results support the use of cerebral atrophy measures from CT in clinical research, as visual ratings showed generally good agreement between CT and MRI, between raters, and between timepoints.

6 Back-To-Drive Project. Assessment of Fitness-To-Drive in Older and Cognitively Impaired Adult Drivers: Adaptation of the on-Road Driving Observation Schedule to Simulation-Based Environments

Objective:Cognitive, motor and sensory deficits associated with aging, and with some neurological conditions such as acquired brain injury, may lead to severe driving performance impairment. While rehabilitation and driver assistance technologies may improve driving performance, the assessment of the actual fitness-to-drive of these people is challenging. Office-based neuropsychological/physical tests are considered insufficient to understand one’s ability to drive. The gold standard is the on-road assessment with dual control cars, superior in ecological validity, but expensive, stressful, and potentially unsafe. Valid, more cost-effective solutions for a safer, more accurate, standardized assessment of fitness-to-drive are currently needed. Modern and sensorized driving simulators offer key advantages, such as the possibility of exposing drivers to several relevant driving scenarios, including hazard situations, and of assessing their driving performance without being physically at risk. However, the extraction and direct interpretation of existing simulator-produced data may require specialized data processing skills or simulation expertise. To overcome this, we have developed an easy-to-use, pencil-and-paper observational instrument. The Sim-DOS is an adaptation of the widely used instrument to assess “natural driving”, the Driving Observation Schedule (DOS; Vlahodimitrakou et al., 2013).Participants and Methods:Via expert consensus, DOS targeted behaviors were adapted to a simulated-based environment (signaling, observation of environment, speed regulation, slow or unsafe reaction, distance interpretation, vehicle/lane positioning), and the Sim-DOS scores calculation (based on errors while doing such behaviors) was adapted from DOS to include hazard situations (HS, 0-100) and free driving (FD, 0-°°) scores. The instrument was then piloted with a sample of 35 older adults, along with the collection of simulator-produced data on number of harsh events and driving speed. Participants drove two consecutive 20-minutes long scenarios, with low and high traffic density (LTD, HTD). In each scenario, there were periods with and without potentially hazard situations.Results:Assessments were performed by two independent trained observers, producing substantial inter-rater reliability (intra-class correlation coefficients above 0.94). Participants (70.7±4.1 years old, 60% male, 46.1±6.7 years of driving experience) were mostly regular drivers (74%). However, psychomotor skills of the majority were compromised, with only one participant being above the 80th percentile in the reaction times test of the national mandatory driving assessment. When exposed to hazard situations, most of the participants (94.1%) did not perform well, independently of traffic density, with average Sim-DOS-HS scores of 87.1±9.7 (out of 100, t-values&gt;7.3, p-values&lt;.05).Compared to LTD scenarios, in HTD scenarios participants drove less smoothly (HTD:0.97±1.24 vs. LTD:0.33±0.58 of harsh events, Z=3.1, p&lt;.05). However, they also drove slower (HTD:82.41±27.43 vs. LTD:103.55±14.61 km/h, t=5.2, p&lt;.05), improving their ability to manage hazard situations, and therefore producing higher than expected Sim-DOS scores (HTD:87.05±10.28). During free driving, participants performed worse under LTD conditions (Sim-DOS-FD scores: HTD:11.68±6.20 vs. LTD:14.40±9.58, t=2.15, p&lt;.05) as they drove at higher speed (HTD:85.01±24.28 vs. LTD:104.70±11.94 km/h, t=5.8, p&lt;.05), although they did it more smoothly (HTD:1.94±3.74 vs. LTD:0.45±0.74 harsh events, Z=2.65, p&lt;.05).Conclusions:Our study provides a validated driving assessment tool for use in driving simulators that will allow a safer, more ecologic, holistic and informative evaluation of the fitness-to-drive of older adults and neurological patients.

Accelerometric Classification of Resting and Postural Tremor Amplitude.

Clinical rating scales for tremors have significant limitations due to low resolution, high rater dependency, and lack of applicability in outpatient settings. Reliable, quantitative approaches for assessing tremor severity are warranted, especially evaluating treatment effects, e.g., of deep brain stimulation (DBS). We aimed to investigate how different accelerometry metrics can objectively classify tremor amplitude of Essential Tremor (ET) and tremor in Parkinson's Disease (PD). We assessed 860 resting and postural tremor trials in 16 patients with ET and 25 patients with PD under different DBS settings. Clinical ratings were compared to different metrics, based on either spectral components in the tremorband or pure acceleration, derived from simultaneous triaxial accelerometry captured at the index finger and wrist. Nonlinear regression was applied to a training dataset to determine the relationship between accelerometry and clinical ratings, which was then evaluated in a holdout dataset. All of the investigated accelerometry metrics could predict clinical tremor ratings with a high concordance (>70%) and substantial interrater reliability (Cohen's weighted Kappa > 0.7) in out-of-sample data. Finger-worn accelerometry performed slightly better than wrist-worn accelerometry. We conclude that triaxial accelerometry reliably quantifies resting and postural tremor amplitude in ET and PD patients. A full release of our dataset and software allows for implementation, development, training, and validation of novel methods.

Are ChatGPT's Free-Text Responses on Periprosthetic Joint Infections of the Hip and Knee Reliable and Useful?

This study aimed to evaluate ChatGPT's performance on questions about periprosthetic joint infections (PJI) of the hip and knee. Twenty-seven questions from the 2018 International Consensus Meeting on Musculoskeletal Infection were selected for response generation. The free-text responses were evaluated by three orthopedic surgeons using a five-point Likert scale. Inter-rater reliability (IRR) was assessed via Fleiss' kappa (FK). Overall, near-perfect IRR was found for disagreement on the presence of factual errors (FK: 0.880, 95% CI [0.724, 1.035], p < 0.001) and agreement on information completeness (FK: 0.848, 95% CI [0.699, 0.996], p < 0.001). Substantial IRR was observed for disagreement on misleading information (FK: 0.743, 95% CI [0.601, 0.886], p < 0.001) and agreement on suitability for patients (FK: 0.627, 95% CI [0.478, 0.776], p < 0.001). Moderate IRR was observed for agreement on "up-to-dateness" (FK: 0.584, 95% CI [0.434, 0.734], p < 0.001) and suitability for orthopedic surgeons (FK: 0.505, 95% CI [0.383, 0.628], p < 0.001). Question- and subtopic-specific analysis revealed diverse IRR levels ranging from near-perfect to poor. ChatGPT's free-text responses to complex orthopedic questions were predominantly reliable and useful for orthopedic surgeons and patients. Given variations in performance by question and subtopic, consulting additional sources and exercising careful interpretation should be emphasized for reliable medical decision-making.

Use of 3-Dimensional Stereophotogrammetry to Detect Disease Progression in Craniofacial Morphea

Objectively determining disease progression in craniofacial morphea (CM) is challenging, as clinical findings of disease activity are often lacking. To evaluate the utility of 3-dimensional (3D) stereophotogrammetry in detecting disease progression in CM over time. This prospective cohort study included 27 pediatric and adult patients with CM from 2 hospitals in Boston (Boston Children's Hospital and Brigham & Women's Hospital) consecutively enrolled from April 1, 2019, to March 1, 2023. Review of 3D stereophotogrammetry images and data analysis occurred from March 1 to April 1, 2023. Clinical and 3D stereophotogrammetry assessments were performed at 2- to 12-month intervals, depending on the clinical context. The 3D stereophotogrammetry images were then qualitatively rated as demonstrating no progression or definitive progression by an expert (board-certified plastic craniofacial surgeon) and nonexpert (board-certified dermatologist) in 3D stereophotogrammetry. In addition, κ coefficients were calculated for interrater reliability. Of 27 patients with CM (19 female; median age, 14 [range, 5-40] years) and 3D stereophotogrammetry images obtained from a minimum of 2 time points (median, 4 [range, 2-10] images) spaced a median of 3 (range, 2-12) months apart, 10 experienced progression of their disease based on clinical assessments performed during the study period. In all cases in which clinical progression was favored, blinded qualitative assessment of 3D stereophotogrammetry images also favored progression with substantial interrater reliability (κ = 0.80 [95% CI, 0.61-0.99]). Furthermore, review of 3D stereophotogrammetry detected occult progression of asymmetry not noted on clinical examination in 3 additional patients. In this prospective cohort study, blinded assessment of sequential 3D stereophotogrammetry images in patients with CM not only corroborated clinical assessment of disease progression but also detected occult progression of facial asymmetry not appreciable on clinical examination alone. Therefore, 3D stereophotogrammetry may serve as a useful adjunct to clinical examination of patients with CM over time. Future investigations are warranted to validate 3D stereophotogrammetry as an outcome measure in CM.

Comparison of teledentistry and on-site clinical diagnosis using the International Caries Detection and Assessment System (ICDAS) in Rawalpindi and Islamabad: a validation study

Dental caries significantly impact global health, particularly in Pakistan, where the prevalence is notably high. This study assessed the diagnostic accuracy of mobile teledentistry in Islamabad and Rawalpindi by employing the International Caries Detection and Assessment System (ICDAS) as a benchmark. This cross-sectional validation study was conducted from July to September 2022 in nine private dental clinics involving 460 participants proficient in basic Android smartphone functions. The diagnostic procedure comprised initial in-person evaluations by dentists, subsequent remote assessments using images captured by smartphones based on verbal instructions, and validations by a third independent dentist. Statistical analyses were performed using sensitivity, specificity, Matthew’s correlation coefficient, and receiver operating characteristic (ROC) curves. Teledentistry achieved a specificity of 89% and a sensitivity of 70%. The positive and negative predictive values were 74.6% and 86%, respectively. The ROC curve analysis, with an area under the curve greater than 0.7, confirmed the high diagnostic efficacy of the method. The Cohen’s kappa statistic reached 0.814, demonstrating substantial interrater reliability, while the diagnostic odds ratio was 18, indicating strong discriminative power. This study confirmed that teledentistry, enabled by the use of smartphone cameras, serves as a reliable method for detecting dental caries, with efficacy comparable to that of conventional clinical examination methods. It significantly extends dental care accessibility, particularly in geographically remote or economically underserved regions, effectively mitigating the logistical challenges associated with traditional dental visits. The findings support the broader implementation of teledentistry, highlighting its potential to enhance mass screening efforts, improve oral health outcomes, and alleviate pressures on healthcare systems across diverse demographic landscapes.

230 Inter-rater reliability assessment of OPTI-3S (criteria for optimising medicines by stopping, stepping down, or switching to safer alternatives)

Abstract Background The OPTI-3S criteria comprise newly developed consensus-based statements identifying potentially inappropriate medications for hospitalised frail older adults. This study assessed the inter-rater reliability of the OPTI-3S core list. Methods Data were collated from ten patient cases in the publicly archived ‘The Irish Longitudinal Study on Ageing’ (TILDA) dataset (Wave 3). Five healthcare professionals assessed medication appropriateness across the cases according to the OPTI-3S statements, indicating the applicable statement(s). Their assessments were compared against a ‘Gold Standard (GS) assessment’ consisting of a combined assessment of a research team member and a clinically experienced academic researcher, who independently assessed the medications’ appropriateness. Concordance between the GS rating and other raters was assessed by kappa statistics. Major discrepancies were identified when a rater applied a statement &amp;gt;2 times more or less frequently than GS [1]. Results The ten patients’ median age was 74 (Interquartile Range: 70–80) years, and 40% were female. The median (IQR) number of medications was 10 (9–12). Cohen’s kappa coefficient ranged from 0.65 to 0.86, indicating substantial to perfect interrater reliability when agreement was measured between the GS rating and any other rater. The overall mean (95% CI) Fleiss kappa coefficient between all raters was 0.67 (0.65,0.69), indicating substantial interrater reliability. The mean Fleiss kappa coefficient (95% CI) between the GS rating and geriatricians was 0.63 (0.60, 0.67). A good Fleiss kappa coefficient of 0.81 (0.75, 0.86) was found between the GS rating and pharmacists. Major discrepancies in applying seven statements were noted between the GS rating and three raters. Three of these statements were misinterpreted, indicating revised wording is needed. Conclusion The OPTI-3S core list shows substantial inter-rater reliability between experts with no previous experience using it. However, further testing with a larger number of clinical cases and raters is required.

Grading Embolization of Middle Meningeal Artery for Chronic Subdural Hematoma.

Embolization of middle meningeal artery (EMMA) is a relatively new treatment for chronic subdural hematoma (CSDH). To date, an objective method that assesses or describes the extent of EMMA for the treatment of CSDH does not exist. Recently, the concept of a novel grading scale for EMMA in patients with CSDH has emerged. However, this has not been applied to a clinical case setting and inter-rater reliability has not yet been studied. The purpose of this study was to validate the grading scale in clinical practice and to assess for inter-rater reliability. We retrospectively examined consecutive patients who underwent EMMA for CSDH. Patients were included if the whole head angiogram from common carotid as well as external carotid arteries before and after EMMA were available in the arterial, capillary as well as venous phases. Two independent readers, each with more than 5 years of experience in independent practice, assessed the angiograms for the grading of EMMA and assigned a score ranging between 0 and 3. The grading score between the two readers were compared using Cohen's Kappa score to assess the inter-rater reliability. In 19 patients, we found that EMMA had no periprocedural morbidity and mortality. The number of cases in each EMMA grading score category are as follows: 0 n =1; 1 n =3; 2 n =1; and 3 n =10. There was substantial inter-rater reliability for the assessment of grading of EMMA (Kappa = 0.74). The novel EMMA grading scheme demonstrated substantial inter-rater reliability and appears promising.