Substantial Inter-rater Reliability Research Articles

Assessing the reliability of pediatric emergency medicine billing code assignment for future consideration as a proxy workload measure.

Prediction of pediatric emergency department (PED) workload can allow for optimized allocation of resources to improve patient care and reduce physician burnout. A measure of PED workload is thus required, but to date no variable has been consistently used or could be validated against for this purpose. Billing codes, a variable assigned by physicians to reflect the complexity of medical decision making, have the potential to be a proxy measure of PED workload but must be assessed for reliability. In this study, we investigated how reliably billing codes are assigned by PED physicians, and factors that affect the inter-rater reliability of billing code assignment. A retrospective cross-sectional study was completed to determine the reliability of billing code assigned by physicians (n = 150) at a quaternary-level PED between January 2018 and December 2018. Clinical visit information was extracted from health records and presented to a billing auditor, who independently assigned a billing code-considered as the criterion standard. Inter-rater reliability was calculated to assess agreement between the physician-assigned versus billing auditor-assigned billing codes. Unadjusted and adjusted logistic regression models were used to assess the association between covariables of interest and inter-rater reliability. Overall, we found substantial inter-rater reliability (AC2 0.72 [95% CI 0.64-0.8]) between the billing codes assigned by physicians compared to those assigned by the billing auditor. Adjusted logistic regression models controlling for Pediatric Canadian Triage and Acuity scores, disposition, and time of day suggest that clinical trainee involvement is significantly associated with increased inter-rater reliability. Our work identified that there is substantial agreement between PED physician and a billing auditor assigned billing codes, and thus are reliably assigned by PED physicians. This is a crucial step in validating billing codes as a potential proxy measure of pediatric emergency physician workload.

Validation of the Acoustic Voice Quality Index and the Acoustic Breathiness Index in the Persian Language

This study aimed to validate the acoustic voice quality index version 03.01 (AVQIv3) and the acoustic breathiness index (ABI) in the Persian language. Six judges assessed the severity degree of the abnormal overall voice quality (hoarseness) and breathiness from continuous speech (cs) and sustained vowel (sv) in 136 voices samples of dysphonic and 45 vocally healthy participants. Firstly, the time length of cs and sv was balanced out for a higher level of ecological validity. Secondly, the concurrent validity and diagnostic accuracy were statistically analyzed with the Spearman rank-order correlation, and the receiver operating characteristics, likelihood ratio, and Youden index. There was substantial inter-rater reliability between judges regarding hoarseness and breathiness. Twenty syllables were identified as the standardized number of syllables for the cs part. Sufficient correlations were found between AVQI and hoarseness (rs =0.74), and between ABI and breathiness (rs =0.74), respectively. The thresholds of 1.63 (sensitivity of 73.2% and specificity of 91%) and 2.97 (sensitivity of 70% and specificity of 87%) yielded the highest level of diagnostic findings for AVQI and ABI, respectively. The AVQIv3 and ABI are reliable indices to quantify the abnormal overall voice quality and breathiness in the Persian language.

Characterizing Camptocormia in Parkinson's Disease Using Muscle Ultrasonography.

Camptocormia (CC) is the forward-bending of the spine of more than 30 degrees that can be found in Parkinson's disease (PD) as a disabling complication. Detection of changes in paraspinal lumbar musculature in CC is of value for choosing treatment strategies. To investigate whether these changes can be detected using muscle ultrasonography (mUSG). Age and sex-matched groups comprised 17 PD patients with CC (seven acute, PD-aCC; 10 chronic PD-cCC), 19 PD patients with no CC, and 18 healthy controls (HC). Lumbar paravertebral muscles (LPM) on both sides were assessed using mUSG by two different raters blinded to the group assignment. Groups were compared with regard to the linear measurements of the muscle thickness as well as semi-quantitative and quantitative (grayscale) analyses of muscle echogenicity using a univariate general linear model. All assessments showed substantial interrater reliability. The PD-cCC group had significantly thinner LPM compared to groups with no CC (PD and HC). Groups of PD-aCC and PD-cCC differed from the groups of no CC in quantitative and semi-quantitative analyses of LPM echogenicity, respectively. Assessment of LPM in PD patients with CC can be reliably performed using mUSG. Also, mUSG may be used as a screening tool to detect CC-related changes in thickness and echogenicity of the LPM in patients with PD.

Accuracy of Assessing Weight Status in Adults by Structured Observation

The assessment of weight status is important in many epidemiological studies, but its direct measurement is not always possible. Self-reported weight and height are often used, although previous research reported low accuracy. This study aimed to test the ability of trained observers to accurately estimate weight status in adults using structured observation. A cross-sectional study was conducted. For each participant, height and weight were estimated in categories, and weight status was recorded using Stunkard’s body figures, by two trained observers. Height and weight were also measured, using standardized procedures. Subjects were classified according to World Health Organization body mass index (BMI) cut-offs from objective measurements and from the BMI assigned to each body figure. Sensitivity, specificity, and likelihood ratios were calculated to assess the accuracy of estimating weight status by observation. Kappa was used to test inter-observer reliability. A total of 127 participants were assessed, 70 women and 57 men, aged between 19 and 89 years (mean ± standard deviation: 50.3 ± 16.3 years). Most participants were overweight or obese (64.3% women; 78.9% men). The sensitivity and specificity of overweight/obesity status identification were 72.8% and 78.4%, respectively. Observers’ gender, participants’ gender, and participants’ age were significantly associated with the estimation of overweight/obesity. The agreement between observers was moderate for BMI estimates (κ = 0.52) but substantial when distinguishing normal weight from overweight/obesity (κ = 0.67). Trained observers were able to distinguish normal weight from overweight/obesity with high sensitivity and specificity, and substantial interrater reliability. This innovative methodology showed potential for improvement through enhanced training techniques. The use of structured observation may be a useful and accurate alternative to self-reported weight status assessment, whenever anthropometric measurement is not achievable.

Development of a Classification System for Live Surgical Feedback

Live feedback in the operating room is essential in surgical training. Despite the role this feedback plays in developing surgical skills, an accepted methodology to characterize the salient features of feedback has not been defined. To quantify the intraoperative feedback provided to trainees during live surgical cases and propose a standardized deconstruction for feedback. In this qualitative study using a mixed methods analysis, surgeons at a single academic tertiary care hospital were audio and video recorded in the operating room from April to October 2022. Urological residents, fellows, and faculty attending surgeons involved in robotic teaching cases during which trainees had active control of the robotic console for at least some portion of a surgery were eligible to voluntarily participate. Feedback was time stamped and transcribed verbatim. An iterative coding process was performed using recordings and transcript data until recurring themes emerged. Feedback in audiovisual recorded surgery. The primary outcomes were the reliability and generalizability of a feedback classification system in characterizing surgical feedback. Secondary outcomes included assessing the utility of our system. In 29 surgical procedures that were recorded and analyzed, 4 attending surgeons, 6 minimally invasive surgery fellows, and 5 residents (postgraduate years, 3-5) were involved. For the reliability of the system, 3 trained raters achieved moderate to substantial interrater reliability in coding cases using 5 types of triggers, 6 types of feedback, and 9 types of responses (prevalence-adjusted and bias-adjusted κ range: a 0.56 [95% CI, 0.45-0.68] minimum for triggers to a 0.99 [95% CI, 0.97-1.00] maximum for feedback and responses). For the generalizability of the system, 6 types of surgical procedures and 3711 instances of feedback were analyzed and coded with types of triggers, feedback, and responses. Significant differences in triggers, feedback, and responses reflected surgeon experience level and surgical task being performed. For example, as a response, attending surgeons took over for safety concerns more often for fellows than residents (prevalence rate ratio [RR], 3.97 [95% CI, 3.12-4.82]; P = .002), and suturing involved more errors that triggered feedback than dissection (RR, 1.65 [95% CI, 1.03-3.33]; P = .007). For the utility of the system, different combinations of trainer feedback had associations with rates of different trainee responses. For example, technical feedback with a visual component was associated with an increased rate of trainee behavioral change or verbal acknowledgment responses (RR, 1.11 [95% CI, 1.03-1.20]; P = .02). These findings suggest that identifying different types of triggers, feedback, and responses may be a feasible and reliable method for classifying surgical feedback across several robotic procedures. Outcomes suggest that a system that can be generalized across surgical specialties and for trainees of different experience levels may help galvanize novel surgical education strategies.

Accuracy and Reliability of Left Atrial Appendage Morphology Assessment by Three-Dimensional Transesophageal Echocardiographic Glass Rendering Modality: A Comparative Study With Computed Tomography

Contrast-enhanced computed tomography is the reference-standard imaging technique to assess left atrial appendage (LAA) morphology. The aim of this study was to evaluate the accuracy and reliability of two-dimensional and new three-dimensional (3D) transesophageal echocardiographic rendering modalities in assessing LAA morphology. Seventy consecutive patients who underwent both computed tomography and transesophageal echocardiography (TEE) were retrospectively enrolled. The traditional LAA morphology classification system (LAAcs; chicken wing, cauliflower, cactus, and windsock) and a new simplified LAAcs based on the LAA bend angle were used for the analysis. LAA morphology was independently assessed by two trained readers using three different modalities: two-dimensional TEE, 3D TEE with multiplanar reconstruction, and a new 3D transesophageal echocardiographic rendering modality with improved transparency (Glass). The new LAAcs and traditional LAAcs were compared in terms of intra- and interrater reliability. With the new LAAcs, two-dimensional TEE was fairly accurate in identifying LAA morphology (κ=0.43, P<.05), with moderate interrater (κ=0.50, P<.05) and substantial intrarater (κ=0.65, P<.005) agreement. Three-dimensional TEE showed higher accuracy and reliability: 3D TEE with multiplanar reconstruction had almost perfect accuracy (κ=0.85, P<.001) and substantial (κ=0.79, P<.001) interrater reliability, while 3D TEE with Glass had substantial accuracy (κ=0.70, P<.001) and almost perfect (κ=0.84, P<.001) interrater reliability. Intrarater agreement was almost perfect for both 3D transesophageal echocardiographic modalities (κ=0.85, P<.001). Accuracy was considerably lower when the traditional LAAcs was used, with 3D TEE with Glass being the most reliable technique (κ=0.75, P<.05). The new LAAcs showed higher inter- and intrarater reliability compared with the traditional LAAcs (interrater, κ=0.85 vs κ=0.49; intrarater, κ=0.94 vs κ=0.68; P<.05). Three-dimensional TEE is an accurate, reliable, and feasible alternative to computed tomography in assessing LAA morphology with the new LAAcs. The new LAAcs shows higher reliability rates than the traditional one.

Hidradenitis Suppurativa: A Cross-sectional Study of Content Quality on TikTok

Introduction: TikTok is a popular social media platform that is sometimes used to share medical information. 1 Creators share their experiences with or provide education regarding medical diagnoses. This study investigated the content quality of TikTok videos regarding hidradenitis suppurativa (HS). Methods: On October 24, 2022, we searched #hidradenitissuppurativa on TikTok and filtered by most liked videos. Two independent reviewers analyzed the videos, excluding those not in English, irrelevant to HS, and duplicates. Videos were scored using the Global Quality Scale (GQS), 2 a one question scale (1-poor quality to 5-excellent quality), and the DISCERN tool, 3 a 16-question analysis (1-low quality to 5-high quality). Results: One-hundred videos were included. The mean GQS score was 2.49 [SD 0.87] and mean DISCERN score was 1.98 [SD 0.52], with substantial interrater reliability (Cohen’s Kappa 0.667). Twenty-six videos (26%) were created by physicians and 74 (74%) were created by non-physicians. Mean GQS and DISCERN scores for physician versus non-physician videos were 3.40 and 2.62, and 2.16 and 1.75, respectively. Both GQS and DISCERN scores were greater for videos created by physicians versus non-physicians (p &lt; 0.0001). Conclusion: Overall, there is a lack of information on treatment benefits, risks, and mechanism of action. Given the mean GQS and DISCERN scores, the average video was of generally poor quality, creating potential for lack of knowledge and misinformation regarding HS. With the growing use of TikTok for educational purposes, improving content quality should be a priority to provide accurate information regarding medical conditions such as HS.

Three-Grade Magnetic Resonance Imaging–Based Gluteus Medius and/or Minimus Tear Classification System Provides Excellent Inter-Rater Reliability

To develop a magnetic resonance imaging (MRI)-based classification system integrating tear characteristics including tear thickness (partial vs full) and tear retraction (less than or greater than 2 cm) for gluteus medius and/or minimus tears and to determine the inter-rater reliability of this MRI-based classification for gluteus medius and/or minimus tears. Patients who underwent primary endoscopic or open repair of gluteus medius and/or minimus tears between 2012 and 2022 were identified to be included in the review of 1.5-T MRI scans. One hundred MRI scans were randomized for review by 2 orthopaedic surgeons and evaluated for tear thickness (partial vs full), extent of retraction, and degree of fatty infiltration according to an applied Goutallier-Fuchs (G-F) classification. Tears were also graded according to the 3-grade MRI-based classification system as follows: grade 1, partial-thickness tears; grade 2, full-thickness tears with less than 2 cm of retraction; grade 3, full thickness with 2 cm or more retraction. Inter-rater reliability was calculated by absolute and relative agreement using Cohen's kappa (κ). Significance was defined by P value <.05. In total, 221 patients were identified, and after application of exclusion criteria and randomization, 100 scans were evaluated. The 3-grade classification system demonstrated high absolute agreement (88%) comparable to the absolute agreement of the G-F classification (67%). The 3-grade classification system demonstrated substantial inter-rater reliability (κ= 0.753), whereas the G-F classification demonstrated moderate inter-rater reliability (κ= 0.489). The proposed 3-grade MRI-based classification system for gluteus medius and/or minimus tears demonstrated substantial inter-rater reliability, comparable with that of the applied G-F classification. It is important to understand how gluteus medius and/or minimus tear characteristics impact postoperative outcomes. The 3-grade MRI-based classification incorporates tear thickness and amount of retraction that can complement previous classification systems to give the provider and patient more information when considering treatment options.

Investigator Global Assessment for impetiginization in atopic dermatitis: development and initial reliability testing.

Secondary infections with Staphylococcus aureus (i.e. impetiginization) are common in patients with atopic dermatitis (AD). However, a scale to assess the severity of impetiginization is lacking. This study aimed to develop and investigate an Investigator Global Assessment (IGA) for impetiginization in AD (‘IGA impetiginization’). Consensus was achieved for a 3-point IGA scale containing a description of clinical signs and extent of the affected area. Substantial inter-rater reliability [intraclass correlation coefficient (ICC) 0.68], good intra-rater reliability (ICC 0.81) and good overall agreement (81%) were found. The IGA impetiginization is a simple scale for the assessment of impetiginization in AD that could be used in clinical practice and research.

P268 Development and validation of a novel composite index for the assessment of endoscopic and histologic disease activity in pouchitis: The Atlantic pouchitis index.

Abstract Background Several instruments have been used in clinical trials of pouchitis to define eligibility criteria and outcome measures but have not been formally validated. A need remains for a validated pouchitis instrument that is suitable for use in novel drug development. We developed and internally validated the novel composite Atlantic pouchitis index (API) for the assessment of endoscopic and histologic disease activity in patients with pouchitis. Methods Paired baseline and week 10 endoscopic videos and histologic images were obtained from 98 patients with chronic antibiotic-refractory pouchitis who participated in a randomised placebo-controlled trial of alicaforsen enema (ClinicalTrials.gov identifier: NCT02525523). For each pair, 4 endoscopists and 3 pathologists scored 59 items. These items included 51 component items from 11 indices (4 endoscopic, 4 histologic, 3 composite) identified in a prior systematic review and 8 additional items identified in a prior Research and Development/University of California Los Angeles appropriateness methodology exercise. The reliability of all measures was assessed using intraclass correlation coefficients (ICCs) and interpreted according to Landis and Koch benchmarks (slight, 0.00-0.20; fair, 0.21-0.40; moderate, 0.41-0.60; substantial, 0.61-0.80; almost perfect, 0.81-1.00). Responsiveness was evaluated using the area under the receiver operating characteristic curve (AUC). The API was developed using linear regression with a 100-mm visual analogue scale (VAS) of pouchitis activity as an anchor. Results Among all measures assessed for reliability and responsiveness, the simple endoscopic score for Crohn’s disease (SES-CD) and the Robarts histopathology index (RHI) were selected for inclusion in the API. The total API score ranges from 0 to 69, with higher scores indicating more severe disease activity, and can be calculated as API = (3 × SES-CD [0 to 12]) + (1 × RHI [0 to 33]). The API demonstrated almost perfect intra-rater reliability (ICC, 0.88 [95% CI: 0.81, 0.92]), substantial inter-rater reliability (ICC, 0.72 [95% CI: 0.60, 0.79]), and a high correlation with the VAS (r = 0.84 [95% CI: 0.76, 0.89]). Index responsiveness in detecting meaningful improvements in disease activity was higher for the API (AUC, 0.95 [95% CI: 0.89, 0.98]) than for existing endoscopic indices (∆AUC, 0.09-0.24; P≤0.005) (Table). Conclusion The novel API is a composite index of endoscopic (SES-CD) and histologic disease activity (RHI) that is reliable and significantly more responsive than existing endoscopic pouchitis scoring systems. The API is a promising tool for use in clinical trials of novel therapies for pouchitis. Further validation is needed in independent datasets.

Inter-Rater Variability in the Evaluation of Lung Ultrasound in Videos Acquired from COVID-19 Patients

Lung ultrasound (LUS) allows for the detection of a series of manifestations of COVID-19, such as B-lines and consolidations. The objective of this work was to study the inter-rater reliability (IRR) when detecting signs associated with COVID-19 in the LUS, as well as the performance of the test in a longitudinal or transverse orientation. Thirty-three physicians with advanced experience in LUS independently evaluated ultrasound videos previously acquired using the ULTRACOV system on 20 patients with confirmed COVID-19. For each patient, 24 videos of 3 s were acquired (using 12 positions with the probe in longitudinal and transverse orientations). The physicians had no information about the patients or other previous evaluations. The score assigned to each acquisition followed the convention applied in previous studies. A substantial IRR was found in the cases of normal LUS (κ = 0.74), with only a fair IRR for the presence of individual B-lines (κ = 0.36) and for confluent B-lines occupying &lt; 50% (κ = 0.26) and a moderate IRR in consolidations and B-lines &gt; 50% (κ = 0.50). No statistically significant differences between the longitudinal and transverse scans were found. The IRR for LUS of COVID-19 patients may benefit from more standardized clinical protocols.

Reliability of foot posture index (FPI-6) for evaluating foot posture in patients with knee osteoarthritis.

Objective: To determine the reliability of FPI-6 in the assessment of foot posture in patients with knee osteoarthritis (KOA). Methods: Thirty volunteers with KOA (23 females, 7 males) were included in this study, assessed by two raters and at three different moments. Inter-rater and test-retest reliability were assessed with Cohen's Weighted Kappa (Kw) and Intraclass Correlation Coefficient (ICC). Bland-Altman plots and respective 95% limits of agreement (LOA) were used to assess both inter-rater and test-retest agreement and identify systematic bias. Moreover, the internal consistency of FPI-6 was assessed by Spearman's correlation coefficient. Results: FPI-6 total score showed a substantial inter-rater (Kw = .66) and test-retest reliability (Kw = .72). The six items of FPI-6 demonstrated inter-rater and test-retest reliability varying from fair to substantial (Kw = .33 to .76 and Kw = .40 to .78, respectively). Bland-Altman plots and respective 95% LOA indicated that there appeared no systematic bias and the acceptable agreement of FPI-6 total score for inter-rater and test-retest was excellent. There was a statistically significant positive correlation between each item and the total score of FPI-6, which indicated that FPI-6 had good internal consistency. Conclusion: In conclusion, the reliability of FPI-6 total score and the six items of FPI-6 were fair to substantial. The results can provide a reliable way for clinicians and researchers to implement the assessment of foot posture in patients with KOA.

Structured Reporting of Head and Neck Sonography Achieves Substantial Interrater Reliability.

Purpose Ultrasound examinations are often criticized for having higher examiner dependency compared to other imaging techniques. Compared to free-text reporting, structured reporting (SR) of head and neck sonography (HNS) achieves superior time efficiency as well as report quality. However, there are no findings concerning the influence of SR on the interrater reliability (IRR) of HNS. Materials and Methods Typical pathologies (n=4) in HNS were documented by video/images by two certified head and neck ultrasound instructors. Consequently, structured reports of these videos/images were created by n=9 senior physicians at departments of otolaryngology or maxillofacial surgery with DEGUM instructors on staff. Reports (n=36) were evaluated regarding overall completeness and IRR. Additionally, user satisfaction was assessed by a visual analog scale (VAS). Results SR yielded very high report completeness (91.8%) in all four cases with a substantial IRR (Fleiss' κ 0.73). Interrater agreement was high at 87.2% with very good user satisfaction (VAS 8.6). Conclusion SR has the potential to ensure high-quality examination reports with substantial comparability and very high user satisfaction. Furthermore, big data collection and analysis are facilitated by SR. Therefore, process quality, workflow, and scientific output are potentially enhanced by SR.

Interrater reliability in assigning a lung ultrasound score

Background and purposeLung ultrasound (LUS) for physiotherapists is an emerging bedside tool. The LUS score of aeration presents as a possible means of assessing and monitoring lung aeration associated with respiratory physiotherapy treatments. There are no studies to date that have assessed the interrater reliability (IRR) of physiotherapists assigning the LUS score of aeration. This study assessed the IRR of assigning the LUS score among adult, mechanically ventilated patients in an intensive care unit with a clinical suspicion of acute lobar atelectasis. MethodsA convenience sample of patients had an LUS performed by a physiotherapist, and images were independently reviewed by two physiotherapists. Each lung zone was assigned an LUS score between 0 and 3 (with 0 being normal aeration and 3 being complete consolidation, presence of effusion, or pneumothorax). IRR was assessed using the kappa statistic. ResultsA total of 1032 LUS images were obtained. Assigning of the LUS across all lung zones demonstrated substantial agreement with kappa 0.685 (95% confidence interval: 0.650, 0.720). Right (0.702 [0.653, 0.751]) and left (0.670 [0.619, 0.721]) lung zones also demonstrated substantial agreement. ConclusionWe found substantial IRR between physiotherapists in assigning the LUS score in a mechanically ventilated adult population in the intensive care unit. Australian New Zealand clinical trials registration numberACTRN12619000783123

A comparison of online and live training of livestock farmers for an on-farm self-assessment of animal welfare

One approach to strengthening the involvement of farmers or stockpersons in the evaluation and improvement of animal welfare is the implementation of an on-farm self-assessment. A valid comparison of the results with reference values, between or within farms, requires that training of the farmers and reliability testing have taken place. We investigated two different training methods (online vs. live) with a total of 146 livestock farmers from farms with dairy cows and calves, beef cattle, sows and suckling piglets, weaners and fattening pigs, laying hens, broiler chickens, and turkeys from all over Germany. Online tests were conducted by assessing photos/videos of each indicator of the assessment scheme to estimate the inter-rater reliability (prevalence-adjusted and bias-adjusted kappa, PABAK). The farmers were requested to provide information on their professional background and rate their motivation to participate in the training and their subjective training success, meaning their confidence in assessing each indicator later on-farm. They evaluated the feasibility of the training and its impact on their views and attitudes. In general, farmers achieved at least substantial inter-rater reliability (PABAK ≥ 0.61) in 86.8% of all initial tests; 13.4% of the tests were repeated once or more times, resulting in a significant improvement of the agreement, with 90.9% of the tests reaching a PABAK ≥ 0.61. However, reliability was higher for indicators with a lower number of score levels. The subjective evaluation of training success was, on average, positive (score = 74.8 out of 100). No effects of the training method or the farmers’ professional background on the inter-rater reliability or the subjective training success were detected. Furthermore, for both methods, farmers moderately agreed that the training had sharpened their views on the animals, encouraged them to implement the assessment on their farm, and made it clear that self-assessment supports animal management. Although the reported costs and time investment for the online training were significantly lower, the effort required for both methods and the ease of integration into the workflow were ranked as similarly acceptable. Overall, both training methods appear feasible for the training of farmers/stockpersons on the assessment of animal-based indicators.

NegotiAct: Introducing a Comprehensive Coding Scheme to Capture Temporal Interaction Patterns in Negotiations

In the present research, we developed NegotiAct, a comprehensive coding scheme for negotiations, comprising 47 mutually exclusive behavioral codes. NegotiAct was derived by systematically integrating (i) 89 extant coding schemes for negotiations, (ii) pertinent findings from negotiation research, and (iii) specific interaction behaviors that were previously not considered in coding schemes for negotiations (e.g., active listening). To facilitate the application of NegotiAct, we designed a coding manual with precise instructions and with definitions and examples for every code. NegotiAct can be customized to address many research questions in experimental settings as well as field research by splitting codes into more specific behaviors. Thereby, differentiated codes can always be traced back to the original codes, preserving comparability across studies and facilitating cumulative research. In combination with interaction analytical methods, NegotiAct enables scholars to detect and investigate specific communication patterns across the negotiation process. As a first empirical validation of NegotiAct, we demonstrate a substantial interrater reliability for 18 videotaped negotiations (κ = .80) and conduct an exploratory validation analysis, studying the relation of multi-issue offers, active listening, and joint gains.

Existing Bowel Preparation Quality Scales Are Reliable in the Setting of Centralized Endoscopy Reading.

Development of bowel preparation products has been based upon colon cleansing rating by a local endoscopist. It is unclear how bowel preparation scales perform when centrally evaluated. To evaluate the reliability of bowel preparation quality scales when assessed by central readers. Four central readers evaluated 52 videos in triplicate, 2weeks apart, during the entire endoscopic procedure (insertion/withdrawal of the colonoscope) and exclusively on colonoscope withdrawal using the Boston Bowel Preparation Scale (BBPS), Chicago Bowel Preparation scale, Harefield Cleansing Scale, Ottawa Bowel Preparation Quality Scale (OBPQS), Aronchick score, a visual analogue scale, and additional items proposed in a modified Research and Development/University of California Los Angeles appropriateness process. Reliability was assessed with intraclass correlation coefficients. Intraclass correlation coefficients (95% confidence interval) for inter-rater reliability of the quality scales ranged from 0.51 to 0.65 (consistent with moderate to substantial inter-rater reliability) during the entire procedure. Corresponding intraclass correlation coefficients for intra-rater reliability ranged from 0.69 to 0.77 (consistent with substantial intra-rater reliability). Reliability was highest in the right colon and lowest in the left colon. No differences were observed in reliability when assessed for the procedure overall (insertion/withdrawal) relative to assessment on withdrawal alone. All five bowel preparation quality scales had moderate to substantial inter-rater reliability. Panelists considered the Aronchick score too simplistic for clinical trials and recognized that assessment of residual fluid in the Ottawa Bowel Preparation Quality Scale was not amenable to central assessment.

Utility of standard diffusion-weighted magnetic resonance imaging for the identification of ischemic optic neuropathy in giant cell arteritis

This study assessed diffusion abnormalities of the optic nerve (ON) in giant cell arteritis (GCA) patients with acute onset of visual impairment (VI) using diffusion-weighted magnetic resonance imaging (DWI). DWI scans of GCA patients with acute VI were evaluated in a case-control study. Two blinded neuroradiologists assessed randomized DWI scans of GCA and controls for ON restricted diffusion. Statistical quality criteria and inter-rater reliability (IRR) were calculated. DWI findings were compared to ophthalmological assessments. 35 GCA patients (76.2 ± 6.4 years; 37 scans) and 35 controls (75.7 ± 7.6 years; 38 scans) were included. ON restricted diffusion was detected in 81.1% (Reader 1) of GCA scans. Localization of ON restricted diffusion was at the optic nerve head in 80.6%, intraorbital in 11.1% and affecting both segments in 8.3%. DWI discerned affected from unaffected ON with a sensitivity, specificity, positive and negative predictive value of 87%/99%/96%/96%. IRR for ON restricted diffusion was κinter = 0.72 (95% CI 0.59–0.86). DWI findings challenged ophthalmologic diagnoses in 4 cases (11.4%). DWI visualizes anterior and posterior ON ischemia in GCA patients with high sensitivity and specificity, as well as substantial IRR. DWI may complement the ophthalmological assessment in patients with acute VI.

Use of artificial intelligence by novice users to enable accurate point-of-care echocardiographic assessment of left ventricular ejection fraction

Abstract Introduction Point-of-care ultrasound (POCUS) is now universal in the hands of non-experts. However, the results are usually binary (normal/abnormal) but not necessarily specific. Artificial intelligence (AI) is currently used by non-experts in different imaging modalities including echocardiography to aid in diagnosis and decision making. Aim To prospectively evaluate whether medical students trained in POCUS and using an AI based assessment tool can accurately evaluate the left ventricular ejection fraction (LVEF) of patients hospitalized in the cardiology department. Methods Nine medical students were trained in a 6-hrs session that included lectures and hands-on practice. Participants used a hand-held ultrasound machine (VScan Extend) equipped with LVivoEF, an AI-based tool that automatically evaluates LVEF. The clips were assessed for LVEF by three methods: visually by the students, students plus the AI-based tool, and experienced cardiologists. All LVEF measurements were compared with the gold-standard (a formal echocardiogram completed within 24-hrs with the Simpson method and LVEF eyeball assessment by two blinded fellowship-trained echocardiographers). Results The study included 88 patients, (aged 58.3±16.3 yrs, mean BMI 28.3±4.4 kg/m2). Comparing LVEF (continuous values) reported by medical students' visual evaluation, medical students plus AI, and cardiologists vs. the echocardiographers values, revealed Pearson correlations of 0.32 (p=0.003), 0.67 (p&amp;lt;0.0001), and 0.73 (p&amp;lt;0.0001), respectively. The agreements between these three evaluation methods and the echocardiographers, assessed at different LVEF categorical thresholds (at 50% and 40%), using kappa coefficient are presented (Figure 1). Conclusion Medical student use of an AI-based tool with a hand-held ultrasound device can improve their LVEF visual assessment to a level of experienced cardiologists. In addition, the use of AI enabled achieving a moderate to substantial inter-rater reliability with echocardiographers' evaluation. This AI tool can be successfully utilized as a decision support tool for POCUS LVEF evaluation by non-experts. Funding Acknowledgement Type of funding sources: None.

Reliability and Measurement Error of Breast Volume Calculation Using Three-Dimensional Surface Imaging.

Background: Breast lymphedema after breast cancer is challenging to quantify. Three-dimensional (3D) surface imaging is one available technique to measure breast volume, however, the measurement properties of available software programs have not been fully determined. The aim of this study was to determine equivalency of measurements with two software programs as well as reliability, standard error of measurement (SEM), and smallest detectable change (SDC). Methods and Results: Retrospective three-dimensional surface imaging (3D-SI) of 100 breasts taken before or after breast conserving surgery for breast cancer were retrieved for reliability analysis. Three assessors followed a standardized measurement technique using two software programs, Vectra® 3D Analysis Module (VAM) and Breast Sculptor®. Mean breast volume was 489.9 ± 206 cc using VAM and 480.1 ± 229.1 cc using Breast Sculptor. Lin's concordance showed poor agreement between programs (0.81-0.88). Measurements using VAM had excellent intra- and inter-rater reliability with SEM = 4.1% for one assessor and 8.7% for multiple assessors. Breast Sculptor also had excellent intra-rater and substantial inter-rater reliability but the SEM was much larger at 14.5% (intra-rater) and 19.1% (inter-rater). The SDC value was lowest for VAM and a single rater with 56 cc indicating a meaningful change beyond measurement error. Conclusion: Breast volume measurements captured with 3D-SI using VECTRA-XT are highly reliable, but the volumes, SEM, and SDC varied between the two software programs. Measurement error was lowest with VAM software. Although the usefulness of VECTRA-XT and VAM software to detect change in breast volume is promising, further solutions to reduce measurement error are required to improve clinical utility to measure breast lymphedema.