Inter-rater Reliability Analysis Research Articles

Ultrafast Brain MRI with Deep Learning Reconstruction for Suspected Acute Ischemic Stroke.

Background Deep learning (DL)-accelerated MRI can substantially reduce examination times. However, studies prospectively evaluating the diagnostic performance of DL-accelerated MRI reconstructions in acute suspected stroke are lacking. Purpose To investigate the interchangeability of DL-accelerated MRI with conventional MRI in patients with suspected acute ischemic stroke at 1.5 T. Materials and Methods In this prospective study, 211 participants with suspected acute stroke underwent clinically indicated MRI at 1.5 T between June 2022 and March 2023. For each participant, conventional MRI (including T1-weighted, T2-weighted, T2*-weighted, T2 fluid-attenuated inversion-recovery, and diffusion-weighted imaging; 14 minutes 18 seconds) and DL-accelerated MRI (same sequences; 3 minutes 4 seconds) were performed. The primary end point was the interchangeability between conventional and DL-accelerated MRI for acute ischemic infarction detection. Secondary end points were interchangeability regarding the affected vascular territory and clinically relevant secondary findings (eg, microbleeds, neoplasm). Three readers evaluated the overall occurrence of acute ischemic stroke, affected vascular territory, clinically relevant secondary findings, overall image quality, and diagnostic confidence. For acute ischemic lesions, size and signal intensities were assessed. The margin for interchangeability was chosen as 5%. For interrater agreement analysis and interrater reliability analysis, multirater Fleiss κ and the intraclass correlation coefficient, respectively, was determined. Results The study sample consisted of 211 participants (mean age, 65 years ± 16 [SD]); 123 male and 88 female). Acute ischemic stroke was confirmed in 79 participants. Interchangeability was demonstrated for all primary and secondary end points. No individual equivalence indexes (IEIs) exceeded the interchangeability margin of 5% (IEI, -0.002 [90% CI: -0.007, 0.004]). Almost perfect interrater agreement was observed (P > .91). DL-accelerated MRI provided higher overall image quality (P < .001) and diagnostic confidence (P < .001). The signal properties of acute ischemic infarctions were similar in both techniques and demonstrated good to excellent interrater reliability (intraclass correlation coefficient, ≥0.8). Conclusion Despite being four times faster, DL-accelerated brain MRI was interchangeable with conventional MRI for acute ischemic lesion detection. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Haller in this issue.

Intra- and inter-rater reliability of muscle and fat thickness measurements obtained using portable ultrasonography in older adults

Ultrasonography (US) is a promising tool for assessing body composition, offering accuracy and accessibility. However, technician skills and equipment characteristics can impact measurement reliability. We aimed to evaluate the intra- and inter-rater reliability of two evaluators using A-mode Portable Ultrasound to measure muscle and fat thickness in Brazilian older adults. Quantitative cross-sectional study carried out with 150 community-dwelling older adults of Campinas-SP-Brazil, recruited in a retirement preparation program and a geriatric outpatient care unit. Scans of nine anatomical points were performed using the portable ultrasound in A-mode (BodyMetrix Pro System BX2000; Livermore, CA), with 6 scans per location and 3 measurements taken by each evaluator. After the selection and analysis of the generated images, muscle and fat thickness were measured. Reliability estimates between measures of the same evaluator (intra-rater) and between evaluators (inter-rater) were assessed using intraclass correlation coefficients (ICC), and differences within and between technicians were assessed using one-way ANOVA. Analyses were carried out for the overall sample and stratified by sex and by nutritional status. Excellent ICC values (>0.90) were found for subcutaneous fat thickness in intra- and inter-rater reliability analyses. The lowest ICC values were observed for deep abdominal fat thickness measured by evaluator 1 (ICC=0.90) and evaluator 2 (ICC=0.87), as well as in the inter-rater analysis (ICC=0.85). Muscle thickness measurements had satisfactory ICC values for triceps, biceps, anterior thigh, and calf, ranging from moderate to good (ICC 0.50-0.90). A portable A-mode ultrasound demonstrates excellent reliability among both intra- and inter-raters for assessing fat thickness but only moderate for muscle thickness in older adults. Hence, this could prompt inquiries regarding the sole reliance on this tool for quantifying muscle mass and examining sarcopenia in older adults. To our knowledge, this study represents the first assessment of reliability for a portable A-mode ultrasound device conducted with older adults.

Algorithms for Reliability Estimate as a Test – Quality Indicator

This study examined the algorithms for reliability estimate as a test-quality indicator. It was discussed along the methods of estimating reliability such as: Test-retest as measures of stability, Equivalent or Alternative –forms reliability as measures of equivalence and stability, while the measures of internal consistency are Split-half, Kuder-Richardson 20 and 21, Coefficient alpha, Hoyt’s analysis of variance, Scorers (Judge) reliability and Inter-rater reliability. Also, using the reliability coefficient as a Test quality indicator was addressed and variables that affect reliability estimate are itemize as test length, test content, test difficulty, item discrimination, group heterogeneity, student motivation, students testwiseness, time limit and security precautions. Therefore, this paper recommends that, in order to demystify the course at any level of our educational system, specialist in the field of Tests, Measurement and Evaluation should be strictly allowed to handle the course and every professional teachers should be abreast to the procedural ways of estimating reliability of test in the classroom examination as a quality indicator in the teachers’ made test.

Tibial Axis-to-Talus Distance: A Clinically Reliable Measurement for Sagittal Translation of the Talus in Total Ankle Arthroplasty.

Sagittal talar translation is an important factor influencing the sagittal alignment of total ankle arthroplasty (TAA). Thus, accurate measurement of sagittal talar translation is crucial. This study proposes a simple method (tibiotalar distance [TTD]) that can quantify talar translation without being affected by the ankle and subtalar joint condition or the talar component position in patients with TAA. We enrolled 280 eligible patients (296 ankles) who underwent primary TAA between 2005 and 2019 and retrospectively reviewed them for sagittal talar translation. The TTD was measured for each patient on weight-bearing lateral ankle radiographs by 3 raters. In addition, we analyzed interrater and intrarater reliability for the TTD method. We found that the TTD method could quantify the talar translation and was not affected by the preoperative condition of the ankle joint surface, subtalar joint pathologies, or the postoperative talar component position. The TTD method showed an excellent intraclass correlation coefficient (> 0.9) in all interrater and intrarater reliability analyses. In the analysis of 157 healthy, unoperated contralateral ankles, we identified that TTD showed a Gaussian distribution (p = 0.284) and a mean of 38.91 mm (normal range, 29.63-48.20 mm). The TTD method is a simple and reliable method that could be applied to patients with TAA to assess the sagittal talar translation regardless of the pre-and postoperative joint condition and implantation status.

Quality Assessment of YouTube Videos on Complementary and Alternative Medicine (CAM) for Cancer Using a Newly Developed Tool.

The global burden of cancer continues to rise and complementary and alternative medicine (CAM) is attracting a lot of interest. However, quality of online information on CAM, particularly on platforms like YouTube, remains questionable. This study aimed to create a comprehensive assessment tool to assess the quality of CAM-related YouTube videos, crucial for informed decision-making in oncology. The assessment tool was developed by adapting existing criteria for website content analysis to video rating. A YouTube search was conducted using German-language terms related to CAM ("complementary medicine (CM) for cancer" and "alternative medicine (AM) for cancer"). In total 25 videos were assessed based on the defined criteria and assigned to five different types of providers (journalism, healthcare organization, hospital or health insurance, independent person, non-medical organization). Statistical analysis was conducted using IBM SPSS Statistics 27. Interrater reliability analysis showed an Intraclass Correlation Coefficient (ICC) of .91, indicating good to excellent agreement. The average video result was of poor quality, with none of the videos meeting all criteria. The videos achieved a mean rating of 38.2 points (SD: 6.5 points; possible range: 20-60 points). Journalism-based videos showed the most views per days, particularly surpassing those from hospitals or health insurance providers (Kruskal-Wallis-Test: z = 3.14, P = .02). However, there was no statistically significant correlation between video quality and the type of provider or interaction indices. Videos retrieved under the search term "CM" generally scored higher in quality than those under "AM" (Mann-Whitney U test: U = 39.5, P = .04). Nonetheless, "CM" videos were less frequently viewed (Mann-Whitney U test: U = 31.0, P = .01). This study, the first of its kind focusing on CAM in cancer care emphasized the challenges in identifying credible sources on social media platforms such as YouTube. The developed assessment tool offers a more detailed evaluation method for health-related videos but requires further refinement and testing. Collaboration between healthcare and media entities is suggested to improve the dissemination of reliable information on platforms like YouTube.

Artificial Intelligence Applied to Chest X-ray: A Reliable Tool to Assess the Differential Diagnosis of Lung Pneumonia in the Emergency Department.

Considering the large number of patients with pulmonary symptoms admitted to the emergency department daily, it is essential to diagnose them correctly. It is necessary to quickly solve the differential diagnosis between COVID-19 and typical bacterial pneumonia to address them with the best management possible. In this setting, an artificial intelligence (AI) system can help radiologists detect pneumonia more quickly. We aimed to test the diagnostic performance of an AI system in detecting COVID-19 pneumonia and typical bacterial pneumonia in patients who underwent a chest X-ray (CXR) and were admitted to the emergency department. The final dataset was composed of three sub-datasets: the first included all patients positive for COVID-19 pneumonia (n = 1140, namely "COVID-19+"), the second one included all patients with typical bacterial pneumonia (n = 500, "pneumonia+"), and the third one was composed of healthy subjects (n = 1000). Two radiologists were blinded to demographic, clinical, and laboratory data. The developed AI system was used to evaluate all CXRs randomly and was asked to classify them into three classes. Cohen's κ was used for interrater reliability analysis. The AI system's diagnostic accuracy was evaluated using a confusion matrix, and 95%CIs were reported as appropriate. The interrater reliability analysis between the most experienced radiologist and the AI system reported an almost perfect agreement for COVID-19+ (κ = 0.822) and pneumonia+ (κ = 0.913). We found 96% sensitivity (95% CIs = 94.9-96.9) and 79.8% specificity (76.4-82.9) for the radiologist and 94.7% sensitivity (93.4-95.8) and 80.2% specificity (76.9-83.2) for the AI system in the detection of COVID-19+. Moreover, we found 97.9% sensitivity (98-99.3) and 88% specificity (83.5-91.7) for the radiologist and 97.5% sensitivity (96.5-98.3) and 83.9% specificity (79-87.9) for the AI system in the detection of pneumonia+ patients. Finally, the AI system reached an accuracy of 93.8%, with a misclassification rate of 6.2% and weighted-F1 of 93.8% in detecting COVID+, pneumonia+, and healthy subjects. The AI system demonstrated excellent diagnostic performance in identifying COVID-19 and typical bacterial pneumonia in CXRs acquired in the emergency setting.

Static Balance Reliability Test with One Leg Stance Test in Knee Osteoarthritis Patients

Introduction: Osteoarthritis (OA) is the most common joint inflammation that often occurs in the knee region and its prevalence increases with age. The prevalence of knee OA is the highest compared to other regions. Static balance is one of the problems in OA patients which is evaluated by the One leg stance test, but the reliability of this measuring instrument for patients with knee OA has not been reported in Indonesia. Research objective: To determine the reliability of the intra rater and inter rater one leg stance test as a measure of static balance in patients with knee osteoarthritis. Materials and methods: The research method used was an observational study with a methodological research approach. There were 58 respondents who signed informed consent and were involved in this study. Results/findings: Intra rater reliability showed a very high internal consistency in both right and left eye open positions (?&gt;0.9) with ICC=0.8-0.9. Whereas OLS in closed eyes shows moderate to low internal consistency and moderate ICC. The results of the inter-rater reliability analysis showed very high internal consistency and ICC for the right and left leg open eyes. Whereas in closed eyes it has internal consistency and moderate to high ICC for the right and left. Conclusion: The ability to maintain static balance in the closed eye position is something that is not easy for patients with knee OA to do, possibly due to pain and also reduced function in the elderly, this is indicated by the low consistency value for OLS in the closed eye position.

Estimating Cognitive Load in a Mobile Personal Health Record Application: A Cognitive Task Analysis Approach.

Mobile health applications that are designed without considering usability criteria can lead to cognitive overload, resulting in the rejection of these apps. To avoid this problem, the user interface of mobile health applications should be evaluated for cognitive load. This evaluation can contribute to the improvement of the user interface and help prevent cognitive overload for the user. In this study, we evaluated a mobile personal health records application using the cognitive task analysis method, specifically the goals, operators, methods, and selection rules (GOMS) approach, along with the related updated GOMS model and gesture-level model techniques. The GOMS method allowed us to determine the steps of the tasks and categorize them as physical or cognitive tasks. We then estimated the completion times of these tasks using the updated GOMS model and gesture-level model. All 10 identified tasks were split into 398 steps consisting of mental and physical operators. The time to complete all the tasks was 5.70 minutes and 5.45 minutes according to the updated GOMS model and gesture-level model, respectively. Mental operators covered 73% of the total fulfillment time of the tasks according to the updated GOMS model and 76% according to the gesture-level model. The inter-rater reliability analysis yielded an average of 0.80, indicating good reliability for the evaluation method. The majority of the task execution times comprised mental operators, suggesting that the cognitive load on users is high. To enhance the application's implementation, the number of mental operators should be reduced.

TikTok Promotes Diet Culture and Negative Body Image Rhetoric: A Content Analysis

ObjectiveThis study aimed to explore the presence of body image and diet culture rhetoric in videos under the hashtag #HealthyLifestyle on TikTok. MethodsThe top 250 videos under #HealthyLifestyle were categorized using a codebook of wellness topics. We conducted descriptive statistics and interrater reliability analysis. Results#HealthyLifestyle videos had high rates of all coded categories, including negative and positive messages about body image and diet culture. Nearly all content with positive connotations was counteracted by coexisting negative messaging. Conclusions and ImplicationsOur findings suggest that content under #HealthyLifestyle contains messaging conflicting with the definition of a healthy lifestyle. Considering the young audience consuming this content, improved nutrition education and health literacy in schools is essential.

A dataset of cognitive ontology for neuroimaging studies of self-reference

Self-reference (or self-referential processing) refers to the cognitive processes underlying self-related information processing. It is widely studied in cognitive neuroscience to better understand the neural basis of self-cognition of human beings. However, does the term “self-reference” mean the same psychological processes across studies? This fundamental question has been largely disregarded and has not received the attention it deserves. To fill the gap, we built an ontological dataset based on neuroimaging studies of self-reference. We searched the literature and screened the articles following a standard protocol. Then, two independent coders extracted data and standardized operationalizations of self-reference on both behavioral and neural levels, resulting in a cognitive ontological dataset for neuroimaging studies of self-reference. This dataset consists of operationalizations of self-reference (in CSV file format) from 66 neuroimaging articles, coordinates data of brain areas activated by self-reference (in BrainMap format), and corresponding codebooks. The inter-rater reliability analysis indicates that the coding process exhibits an exceptional level of quality. Compared with automatic meta-analytical platforms, i.e., Neurosynth, the current dataset provides a fine-grained granularity in article selection, which allows the comparison of brain regions activated by different operationalizations of self-reference. This dataset lays a foundation for the understanding of neural mechanisms underlying self-cognition. It may also facilitate the study of cognitive ontology by serving as an exemplary model for the creation of similar metascience datasets.

Ultrasound measurement of subsynovial connective tissue thickness in the carpal tunnel: An intrarater/interrater reliability and agreement study

ObjectivesNon-inflammatory thickening of the subsynovial connective tissue (SSCT) in the carpal tunnel is commonly found in subjects with carpal tunnel syndrome (CTS), and quantification may shed light on CTS pathogenesis. To date, information on the reliability of ultrasound quantification of SSCT is scarce. Therefore, we investigated intrarater and interrater reliability/agreement for ultrasound quantification of SSCT thickness in subjects with and without CTS, and predictors for tissue thickness. Material and methodsTwo investigators quantified SSCT thickness and thickness ratio on ultrasound in 16 healthy subjects (age, 24–65 years; 16 left/14 right wrists) and 17 subjects with CTS (age, 37–83 years; 14 left/14 right wrists). Intra- and inter-rater reliability/agreement were assessed on intraclass correlation coefficients, standard error of measurement and minimal detectable change. A mixed-effects model was used to evaluate potential predictors for SSCT thickness. ResultsIntra- and inter-rater reliability analysis showed good to excellent intraclass correlation coefficients in both groups, ranging from 0.772 to 0.965. The maximum percentage standard error of measurement was 8%. The maximum minimal detectable change was 14% within raters, and 20% between raters. Both intra- and inter-rater reliability values for thickness ratio were poor. Presence of CTS (ß = 0.180; p = 0.015) correlated positively with SSCT thickness. ConclusionsUltrasound is a reliable method for quantification of SSCT thickness, but not for thickness ratio. Presence of CTS correlates positively with SSCT thickness.

Intrarater and Interrater Reliability of Radiographic Characteristics in Skeletally Immature Patients With Anterior Cruciate Ligament Tears: A PLUTO Study Group Reliability Study.

Radiographic measurements of limb alignment in skeletally immature patients with anterior cruciate ligament (ACL) tears are frequently used for surgical decision-making, preoperative planning, and postoperative monitoring of skeletal growth. However, the interrater and intrarater reliability of these radiographic characteristics in this patient population is not well documented. Excellent reliability across 4 raters will be demonstrated for all digital measures of length, coronal plane joint orientation angles, mechanical axis, and tibial slope in skeletally immature patients with ACL tears. Cohort study (diagnosis). Three fellowship-trained orthopaedic surgeons and 1 medical student performed 2 rounds of radiographic measurements on digital imaging (lateral knee radiographs and long-leg radiographs) of skeletally immature patients with ACL tears. Intrarater and interrater reliability for continuous radiographic measurements was assessed with intraclass correlation coefficients (ICCs) across 4 raters with 95% CIs for affected and unaffected side measurements. Interrater reliability analysis used an ICC (2, 4) structure and intrarater reliability analysis used an ICC (2, 1) structure. A weighted kappa coefficient was calculated for ordinal variables along with 95% CIs for both interrater and intrarater reliability. Agreement statistic interpretations are based on scales described by Fleiss, and Cicchetti and Sparrow: <0.40, poor; 0.40 to 0.59, fair; 0.60 to 0.74, good; and >0.74, excellent. Radiographs from a convenience sample of 43 patients were included. Intrarater reliability was excellent for nearly all measurements and raters. Interrater reliability was also excellent for nearly all reads for all measurements. Radiographic reliability of long-leg radiographs and lateral knee x-rays in skeletally immature children with ACL tears is excellent across nearly all measures and raters and can be obtained and interpreted as reliable and reproducible means to measure limb length and alignment. Level III.

Reliability and Validity of The Fullerton Advanced Balance Scale in People with Multiple Sclerosis

PurposeThis study was conducted to evaluate the reliability and validity of the Fullerton Advanced Balance Scale (FAB) in people with Multiple Sclerosis (PwMS). MethodsA total of 65 people with multiple sclerosis, Expanded Disability Status Scale (EDSS) ranging from 1 to 5.5, were included in the study. Test-retest reliability, intra-rater, inter-rater reliability, and internal consistency (item-total score correlation, Cronbach's alpha coefficient) were investigated to examine the reliability of FAB. In the intra-rater and inter-rater reliability analysis, the FAB application of 34 patients, whose initial evaluation was gathered, was video-recorded and re-watched by two physiotherapists at different times and scored. For the Validity of FAB, concurrent validity with criterion validity; construct validity with hypothesis testing were calculated. Convergent validity was assessed for correlations with EDDS, Dynamic Gait Index (DGI), and Timed Up and Go Test (TUG). ResultsTest-retest reliability of FAB Intraclass Correlation Coefficient (ICC) values was excellent (ICC= 0.994, p < 0.001). While the intra-rater reliability (ICC=0.986, p < 0.001) and inter-rater reliability (ICC=0.985, p < 0.001) of the FAB were calculated at an excellent level. Cronbach's alpha value was determined to perfect correlation. (Cronbach's alpha coefficient: 0.929). FAB had an excellent correlation with BBS (0.919 (p < 0.001). For convergent validity of FAB, EDSS (r=-0.885, p < 0.001), TUG (r=-0.833, p < 0.001), and DGI (r = 0.916, p < 0.001), it was determined that the scale had convergent validity. ConclusionThe FAB proved to be a reliable and valid in PwMS. The study showed that the FAB could be applied regardless of the physiotherapists' clinical experiences. It has been determined that the scale can be used in PwMS with a wide EDSS score. Considering the inter-rater and intra-rater reliability results, it is thought that the FAB is also suitable for use in the online environment.

Shoulder Range of Motion Measurement Using Inertial Measurement Unit-Concurrent Validity and Reliability.

This study aimed to evaluate the reliability of the RSQ Motion sensor and its validity against the Propriometer and electronic goniometer in measuring the active range of motion (ROM) of the shoulder. The study included 15 volunteers (mean age 24.73 ± 3.31) without any clinical symptoms with no history of trauma, disease, or surgery to the upper limb. Four movements were tested: flexion, abduction, external and internal rotation. Validation was assessed in the full range of active shoulder motion. Reliability was revised in full active ROM, a fixed angle of 90 degrees for flexion and abduction, and 45 degrees for internal and external rotation. Each participant was assessed three times: on the first day by both testers and on the second day only by one of the testers. Goniometer and RSQ Motion sensors showed moderate to excellent correlation for all tested movements (ICC 0.61-0.97, LOA < 23 degrees). Analysis of inter-rater reliability showed good to excellent agreement between both testers (ICC 0.74-0.97, LOA 13-35 degrees). Analysis of intra-rater reliability showed moderate to a good agreement (ICC 0.7-0.88, LOA 22-37 degrees). The shoulder internal and external rotation measurement with RSQ Motion sensors is valid and reliable. There is a high level of inter-rater and intra-rater reliability for the RSQ Motion sensors and Propriometer.

An Exploration of the Risk Factors, Severity, and Sequelae of Vocal Fold Hemorrhage in a Population of Voice Patients Before and After Diagnosis

BackgroundVocal fold hemorrhage (VFH) is the rupture (usually acute) of a blood vessel within the true vocal fold. The long-term sequelae of VFH on the mucosal wave (MW) and glottic gap on video stroboscopy remain understudied. The primary objective of this study was to investigate the short-term and long-term consequences of VFH through measured and rated analysis of the mucosal wave and glottic gap. MethodsThe presence of VFH and its extent (limited/moderate vs. extensive VFH) were identified. The primary outcome of this study was mucosal wave, which was assessed on an ordinal scale by three blinded raters pre and posthemorrhage. Only patients who had undergone strobovideolaryngoscopy before sustaining VFH were included. Mucosal wave and glottic gap also were measured using image pixel analysis using the open-access tool, ImageJ (NIH, Bethesda, MD). ResultsTwenty-three subjects were included in this study (mean age 39.78 ± 15.54). Intra-rater reliability for MW ratings was 81.48% ± 6.150% (minimum 77.78%) for all evaluators (κ = 0.519 [0.267–0.772], P < 0.001). Inter-rater reliability analysis revealed 75.56% agreement between evaluators (κ = 0.524 [0.425–0.623], P < 0.001). MWMeasured extrapolated from ImageJ methodology correlated significantly with MWRated (n = 70, r = 0.448, P < 0.001). ΔMWMeasured from baseline to follow-up evaluation were compared for both the initial follow-up visit (FU1) and the second follow-up visit (FU2) [−4.135 ± 31.01 vs. 36.50 ± 39.97, P = 0.025]. Hence, ΔMWMeasured was significantly better by FU2 than FU1, with the larger positive change from baseline representing a greater improvement in the measured mucosal wave. Additionally, there were significant differences in ΔMWRated between those with limited/moderate VFH and those with extensive VFH at FU1. Duration of absolute voice rest correlated significantly with time to VFH resolution. Long-term change in mucosal wave after hemorrhage was assessed using both ΔMWMeasured and MWRated. Based on ΔMWRater, 35.0% of subjects demonstrated ongoing and worse mucosal wave restriction compared to baseline at their most recent follow-up visit. Based on ΔMWMeasured, 50.0% of the subjects showed ongoing and worse mucosal wave restriction compared to baseline at their most recent follow-up visit. ConclusionOverall long-term restrictions in MW after hemorrhage were present in 35.0% of the subjects based on ratings and 50.0% of the patients based on the measured MW using ImageJ, demonstrating the importance of ongoing study into this pathology and how to prevent it, especially in PVU and professional singers. Patients presenting with extensive hemorrhage were at risk for more prominent, detectable changes in mucosal wave compared to those with limited/moderate hemorrhage in the short-term, defined by a mean FU time of 3 months, but not long-term, characterized by a mean FU time 6 months or greater. Whether the severity of VFH is a true indicator of mucosal wave alterations requires additional study, as does the reliability and validity of ImageJ mucosal wave and glottic gap measurement techniques.

The voice characterisation checklist: psychometric properties of a brief clinical assessment of voices as social agents.

There is growing interest in tailoring psychological interventions for distressing voices and a need for reliable tools to assess phenomenological features which might influence treatment response. This study examines the reliability and internal consistency of the Voice Characterisation Checklist (VoCC), a novel 10-item tool which assesses degree of voice characterisation, identified as relevant to a new wave of relational approaches. The sample comprised participants experiencing distressing voices, recruited at baseline on the AVATAR2 trial between January 2021 and July 2022 (n = 170). Inter-rater reliability (IRR) and internal consistency analyses (Cronbach's alpha) were conducted. The majority of participants reported some degree of voice personification (94%) with high endorsement of voices as distinct auditory experiences (87%) with basic attributes of gender and age (82%). While most identified a voice intention (75%) and personality (76%), attribution of mental states (35%) to the voice ('What are they thinking?') and a known historical relationship (36%) were less common. The internal consistency of the VoCC was acceptable (10 items, α = 0.71). IRR analysis indicated acceptable to excellent reliability at the item-level for 9/10 items and moderate agreement between raters' global (binary) classification of more vs. less highly characterised voices, κ = 0.549 (95% CI, 0.240-0.859), p < 0.05. The VoCC is a reliable and internally consistent tool for assessing voice characterisation and will be used to test whether voice characterisation moderates treatment outcome to AVATAR therapy. There is potential wider utility within clinical trials of other relational therapies as well as routine clinical practice.

Performance of the National Institute of Infectious Diseases disability scale in HTLV-1-associated myelopathy/tropical spastic paraparesis.

HTLV-1-associated myelopathy/tropical spastic paraparesis (HAM/TSP) is a chronic disabling disease. However, there is a lack of an adequate and specific health measurement instrument validated and with good performance to assess their degree of physical disability. This led us to carry out this study and to evaluate the performance of Fiocruz's National Institute of Infectious Diseases (IDS) disability scale, a specific instrument for HAM/TSP. Ninety-two HAM/TSP patients participated in the study. One researcher applied the IDS, IPEC scale, Disability Status Scale (DSS), Expanded DSS (EDSS), Osame scale, Beck Depression Inventory, and the WHOQOL-BREF questionnaire. In parallel, blindly, and separately, other researchers applied the IDS. An inter-rater reliability analysis of the IDS, correlation analysis with the other scales, and depression and quality of life questionnaires were performed. The applicability of the IDS was also evaluated. The IDS showed high reliability in all scores. The inter-rater reliability test for the total IDS score was 0.94 (0.82-0.98) on its four dimensions. The scale adequately indicated the different degrees of disability, presenting a distribution similar to normal. There was a high correlation with the other scales (Spearman coefficients > 0.80, p < 0.001). The scale had good acceptance among users and a short application time. IDS for HAM/TSP was reliable, consistent, easy, and fast to use. It can be used for both prospective evaluations and clinical trials. The present study supports the IDS as a valid instrument to measure disability in patients with HAM/TSP compared to previously used scales.

Occupation-Centered Intervention Assessment in Assignments: A Tool for Explicit Use of Occupation Throughout the Curriculum

Abstract Date Presented 04/22/2023 The purpose of this study was to demonstrate the use of the Occupation-Centered Intervention Assessment (OCIA) in student assignments to develop an improved understanding of occupation-centered practice. In addition, the researchers utilized statistical analysis of interrater reliability as a measure of student learning. The students demonstrated good rater agreement for the OCIA, and the researchers were able to identify areas of focus for future instruction regarding occupation-centered practice. Primary Author and Speaker: Kelly Lynn Little Additional Authors and Speakers: Vanessa Jewell Contributing Authors: Yongyue Qi

#4551 DEVELOPMENT OF AN X-RAY–BASED BONE OXALOSIS GRADING SCALE TO ASSESS OXALATE ACCUMULATION IN PATIENTS WITH PRIMARY HYPEROXALURIA TYPE 1

Abstract Background and Aims Primary hyperoxaluria type 1 (PH1) is a rare genetic disease in which hepatic oxalate overproduction can lead to kidney stones, nephrocalcinosis, kidney failure, and systemic oxalosis, a condition in which calcium oxalate is deposited in various tissues, including bone [1]. Radiological signs of bone oxalosis include findings such as dense metaphyseal bands and coarse trabeculation [2]. No scale exists to grade bone oxalosis severity using X-rays. Method An X-ray grading scale to evaluate systemic oxalosis in specific bones was developed based on expert opinion. Scores on individual items ranged from 0–4, except for ribs and spine, which ranged from 0–2. Higher values represent more advanced oxalosis. To validate the scale, 85 X-ray images from 5 pediatric patients with PH1 who had developed bone oxalosis were collected from charts at Shaare Zedek Medical Center and de-identified [2]. Two blinded, independent raters evaluated each X-ray twice and assigned a grade to each applicable item on the scale. Inter-rater and intra-rater reliability analyses were conducted using the weighted Cohen's kappa with asymptotic 95% confidence intervals (CIs) of the estimate and interpreted as proposed in the literature [3]. Total weighted kappa estimates were generated by pooling all observed ratings across all items. Results Total overall inter-rater (0.83 [95% CI: 0.79, 0.87]) and intra-rater (0.95 [0.93, 0.97]) kappa estimates demonstrated almost perfect agreement. Overall inter-rater kappa estimates were &amp;gt;0.8 to 1.0 (almost perfect agreement) for the left hand/wrist, left hip, left knee (femur), and left humerus; &amp;gt;0.6 to 0.8 (substantial agreement) for the right hip, right knee (tibia), right humerus, spine, and ribs; and &amp;gt;0.4 to 0.6 (moderate agreement) for the right knee (femur) and right knee (fibula). The overall inter-rater kappa estimate for the left knee (fibula) demonstrated poor agreement (−0.08 [−0.27, 0.10]). Overall inter-rater kappa estimates for the right hand/wrist and left knee (tibia) were considered unreliable due to lack of variability in the data, and the standard errors were not estimable. Calculable overall intra-rater kappa estimates were &amp;gt;0.8 to 1.0 (almost perfect agreement) for the spine and &amp;gt;0.6 to 0.8 (substantial agreement) for the right knee (femur) and right knee (fibula). Most other overall intra-rater kappa estimates could not be calculated because the kappas for one or both raters lacked variability. In these instances, intra-rater kappa estimates for the first and second raters demonstrated moderate to almost perfect agreement (&amp;gt;0.4 to 1.0; left hand/wrist, right hip, left hip, left knee [femur], right knee [tibia], left knee [tibia], right humerus, left humerus, and ribs). For the left knee (fibula), intra-rater kappa estimates for the first and second raters were −0.14 (−0.34, 0.05) and 1.00, respectively, and for the right hand/wrist, they were 1.00 and 0.00. Conclusion We developed a novel X-ray–based bone oxalosis grading scale for patients with PH1. Total overall weighted kappa estimates for inter-rater and intra-rater reliability demonstrated almost perfect strength of agreement. Most individual items demonstrated reliable kappa estimates. The right and left knee (fibula) were removed from the scale due to poor reliability.

Enhancing Nonverbal Communication Through Virtual Human Technology: Protocol for a Mixed Methods Study.

Communication is a critical component of the patient-provider relationship; however, limited research exists on the role of nonverbal communication. Virtual human training is an informatics-based educational strategy that offers various benefits in communication skill training directed at providers. Recent informatics-based interventions aimed at improving communication have mainly focused on verbal communication, yet research is needed to better understand how virtual humans can improve verbal and nonverbal communication and further elucidate the patient-provider dyad. The purpose of this study is to enhance a conceptual model that incorporates technology to examine verbal and nonverbal components of communication and develop a nonverbal assessment that will be included in the virtual simulation for further testing. This study will consist of a multistage mixed methods design, including convergent and exploratory sequential components. A convergent mixed methods study will be conducted to examine the mediating effects of nonverbal communication. Quantitative (eg, MPathic game scores, Kinect nonverbal data, objective structured clinical examination communication score, and Roter Interaction Analysis System and Facial Action Coding System coding of video) and qualitative data (eg, video recordings of MPathic-virtual reality [VR] interventions and student reflections) will be collected simultaneously. Data will be merged to determine the most crucial components of nonverbal behavior in human-computer interaction. An exploratory sequential design will proceed, consisting of a grounded theory qualitative phase. Using theoretical, purposeful sampling, interviews will be conducted with oncology providers probing intentional nonverbal behaviors. The qualitative findings will aid the development of a nonverbal communication model that will be included in a virtual human. The subsequent quantitative strand will incorporate and validate a new automated nonverbal communication behavior assessment into the virtual human simulation, MPathic-VR, by assessing interrater reliability, code interactions, and dyadic data analysis by comparing Kinect responses (system recorded) to manually scored records for specific nonverbal behaviors. Data will be integrated using building integration to develop the automated nonverbal communication behavior assessment and conduct a quality check of these nonverbal features. Secondary data from the MPathic-VR randomized controlled trial data set (210 medical students and 840 video recordings of interactions) were analyzed in the first part of this study. Results showed differential experiences by performance in the intervention group. Following the analysis of the convergent design, participants consisting of medical providers (n=30) will be recruited for the qualitative phase of the subsequent exploratory sequential design. We plan to complete data collection by July 2023 to analyze and integrate these findings. The results from this study contribute to the improvement of patient-provider communication, both verbal and nonverbal, including the dissemination of health information and health outcomes for patients. Further, this research aims to transfer to various topical areas, including medication safety, informed consent processes, patient instructions, and treatment adherence between patients and providers. DERR1-10.2196/46601.