Inter-rater Reliability Values Research Articles

Workflow for detecting biomedical articles with underlying open and restricted-access datasets.

To monitor the sharing of research data through repositories is increasingly of interest to institutions and funders, as well as from a meta-research perspective. Automated screening tools exist, but they are based on either narrow or vague definitions of open data. Where manual validation has been performed, it was based on a small article sample. At our biomedical research institution, we developed detailed criteria for such a screening, as well as a workflow which combines an automated and a manual step, and considers both fully open and restricted-access data. We use the results for an internal incentivization scheme, as well as for a monitoring in a dashboard. Here, we describe in detail our screening procedure and its validation, based on automated screening of 11035 biomedical research articles, of which 1381 articles with potential data sharing were subsequently screened manually. The screening results were highly reliable, as witnessed by inter-rater reliability values of ≥0.8 (Krippendorff's alpha) in two different validation samples. We also report the results of the screening, both for our institution and an independent sample from a meta-research study. In the largest of the three samples, the 2021 institutional sample, underlying data had been openly shared for 7.8% of research articles. For an additional 1.0% of articles, restricted-access data had been shared, resulting in 8.3% of articles overall having open and/or restricted-access data. The extraction workflow is then discussed with regard to its applicability in different contexts, limitations, possible variations, and future developments. In summary, we present a comprehensive, validated, semi-automated workflow for the detection of shared research data underlying biomedical article publications.

Intra- and Inter-Rater Reliability of Magnetic Resonance Imaging Measurements of Supraspinatus Muscle Thickness, Acromiohumeral Distance, and Coracohumeral Distance in Patients with Shoulder Pain

Aim: The aim of this study was to evaluate the intra- and inter-rater reliability coefficients of the supraspinatus muscle thickness (SMT ), acromiohumeral distance (AHD), and coracohumeral distance (CHD) in patients with shoulder pain. Materials and Methods: This retrospective study included 80 patients who presented with complaints of shoulder pain at a single centre between January 2023 and January 2024. The supraspinatus muscle thickness, acromiohumeral distance, and coracohumeral distance measurements were obtained one week apart by two observers. The intraclass correlation coefficient (ICC), minimum detectable change, and standard error of measurement were subsequently calculated. Results: AHD intra-rater reliability results evaluated on coronal sections were excellent (Rater 1, ICC=0.96; 95% CI, 0.94-0.97; Rater 2 ICC=0.75; 95% CI, 0.61-0.84). AHD intra-rater reliability results evaluated on sagittal sections were excellent (Rater 1, ICC=0.94; 95% CI; 0.91-0.96; Rater 2 ICC=0.77; 95% CI, 0.64-0.85). The results of CHD measured using coronal sections were good (Rater 1, ICC=0.85; 95% CI, 0.76-0.90; Rater 2 ICC=0.82; 95% CI, 0.71-0.88). SMT measurement results measured using coronal sections were excellent (Rater 1, ICC=0.98; CI, 0.98-0.99; Rater 2, ICC=0.89; 95% CI, 0.82-0.93).The inter-rater reliability values were good for AHD-Coronal (ICC= 0.75; 95% CI, 0.61-0.84) and AHD-Sagittal (ICC=0.86; 95% CI, 0.79-0.91), were fair for CHD evaluated on coronal sections (ICC=0.74; %95 CI, 0.58-0.83), and were excellent for SMT evaluated on coronal sections (ICC=0.92; 95% CI, 0.87-0.95). Conclusion: These results suggest that magnetic resonance imaging measurements of the supraspinatus muscle thickness and acromiohumeral distance can be reliable and consistent for the clinical evaluation of patients with shoulder pain. Keywords: Reliability, shoulder measurement, magnetic resonance imaging

Digital Pathology in the Detection of Infectious Microorganisms: An Evaluation of Its Strengths and Weaknesses Across a Panel of Immunohistochemical and Histochemical Stains Routinely Used in Diagnostic Surgical Pathology.

The diagnosis of some infectious diseases requires their identification in tissue specimens. As institutions adopt digital pathology for primary diagnosis, the limits of microorganism detection from digital images must be delineated. To assess the reliability of microorganism detection from digitized images of histochemical and immunohistochemical stains commonly used in pathology. Original glass slides from 620 surgical pathology cases evaluated for the presence of infectious microorganisms were digitized. Immunohistochemical stains included those for herpes simplex virus (n = 100), cytomegalovirus (n = 100), Helicobacter pylori (n = 100), and spirochetes (n = 80). Histochemical stains included mucicarmine for Cryptococcus spp (n = 20), Grocott methenamine silver for fungi (n = 100), Giemsa for H pylori (n = 100), and Ziehl-Neelsen for acid-fast bacilli (n = 20). The original diagnosis based on the glass slides was regarded as the reference standard. Six pathologists reviewed the digital images. Digital review was generally associated with high (ie, ≥90%) specificity and positive predictive value owing to a low percentage of false positive reads, whereas a high percentage of false negatives contributed to low sensitivity and negative predictive value for many stains. Fleiss κ showed substantial interobserver agreement in the interpretation of Grocott methenamine silver and immunostains for herpes simplex virus, H pylori, and cytomegalovirus; moderate agreement for spirochete, Ziehl-Neelsen, and mucicarmine; and poor agreement for Giemsa. Digital immunohistochemistry generally outperforms histochemical stains for microorganism detection. Digital interpretation of Ziehl-Neelsen and mucicarmine stains is associated with low scores for interrater reliability, accuracy, sensitivity, and negative predictive value such that it should not substitute for conventional review of glass slides.

Spine surgeon versus AI algorithm full-length radiographic measurements: a validation study of complex adult spinal deformity patients.

Spinal measurements play an integral role in surgical planning for a variety of spine procedures. Full-length imaging eliminates distortions that can occur with stitched images. However, these images take radiologists significantly longer to read than conventional radiographs. Artificial intelligence (AI) image analysis software that can make such measurements quickly and reliably would be advantageous to surgeons, radiologists, and the entire health system. Institutional Review Board approval was obtained for this study. Preoperative full-length standing anterior-posterior and lateral radiographs of patients that were previously measured by fellowship-trained spine surgeons at our institution were obtained. The measurements included lumbar lordosis (LL), greatest coronal Cobb angle (GCC), pelvic incidence (PI), coronal balance (CB), and T1-pelvic angle (T1PA). Inter-rater intra-class correlation (ICC) values were calculated based on an overlapping sample of 10 patients measured by surgeons. Full-length standing radiographs of an additional 100 patients were provided for AI software training. The AI algorithm then measured the radiographs and ICC values were calculated. ICC values for inter-rater reliability between surgeons were excellent and calculated to 0.97 for LL (95% CI 0.88-0.99), 0.78 (0.33-0.94) for GCC, 0.86 (0.55-0.96) for PI, 0.99 for CB (0.93-0.99), and 0.95 for T1PA (0.82-0.99). The algorithm computed the five selected parameters with ICC values between 0.70 and 0.94, indicating excellent reliability. Exemplary for the comparison of AI and surgeons, the ICC for LL was 0.88 (95% CI 0.83-0.92) and 0.93 for CB (0.90-0.95). GCC, PI, and T1PA could be determined with ICC values of 0.81 (0.69-0.87), 0.70 (0.60-0.78), and 0.94 (0.91-0.96) respectively. The AI algorithm presented here demonstrates excellent reliability for most of the parameters and good reliability for PI, with ICC values corresponding to measurements conducted by experienced surgeons. In future, it may facilitate the analysis of large data sets and aid physicians in diagnostics, pre-operative planning, and post-operative quality control.

Adaptation and reliability of tests of functional mobility and muscle strength using telehealth for older people with dementia

Objective: Adapt functional mobility and muscle strength tests for older people with dementia to be performed remotely in the home environment; determine intra-rater and inter-rater reliability of functional mobility and strength tests for older people with dementia; and correlate the performance on the physical tests executed remotely and the Timed Up and Go (TUG) test, which was validated remotely in healthy older people. Methods: We included 43 people with dementia. The Short Physical Performance Battery (SPPB), TUG, TUG Dual Task (TUG-DT), and 30-Second Sit-to-Stand Test (SST30s) were performed through videocalls. Test performance was documented to facilitate the subsequent determination of inter- and intra-rater reliability. The intra-rater reliability was determined at a 2-week interval from the real-time videocall to analysis of the recording. Results: Intra-rater reliability ranged from good to excellent for all tests performed (ICC 0.862–1.000). Inter-rater reliability values were fair for gait speed (ICC = 0.223), moderate for some of the SPPB items (ICC = 0.706–0.801), good for the SPPB two-foot balance (ICC = 0.860) and the SPPB total score (ICC = 0.831), and excellent for the rest of the tests. Conclusion: Functional mobility and muscle strength tests proved reliable and feasible for remote, at-home assessments of older people with dementia when assisted by their respective caregivers, providing an additional alternative for assessment of this patient population.

Validity and reliability of selective control of upper extremity scale (SCUES) in patients with chronic stroke

Purpose The aim of this study is to investigate the validity and reliability of the Selective Control of Upper Extremity Scale (SCUES) in patients with stroke. Materials and Methods Forty-two patients with stroke aged 18–75 years, were included in the study. Patients were video-recorded while SCUES was administered. The videos were scored to determine the intrarater and inter-rater reliability. Fugl Meyer Assessment of Upper Extremity (FMA-UE), Box and Block test (BBT) and Brunnstrom Stages of Motor Recovery were measured to evaluate validity of SCUES. Spearman correlation analysis was used to assess the validity of SCUES. Intraclass correlation coefficient (ICC), Kappa (κ) and weighted Kappa (κw) were calculated to determine intrarater and inter-rater reliability. Results There were significant positive high correlations between SCUES and FMA-UE and BBT and Brunnstrom upper extremity proximal and distal values (rho = 0.944, p = 0.01; rho = 0,875, p = 0.01; rho = 0.84, p = 0.01, rho = 0.82, p = 0.01; respectively) showing validity of SCUES. The ICC value of inter-rater reliability of SCUES was 0,99 (%95 CI: 0,989-0,997, p = 0,001) showing excellent reliability. κ and κ w values for inter-rater and intrarater reliability of individual SCUES items were above 0.7 indicating excellent reliability. ICC of SCUES and FMA-UE indicated excellent intrarater reliability (ICC = 0,99; %95 CI: 0,989–0,997, p = 0,001; ICC = 0.943; %95 CI: 0.9–0.97, p = 0,0001, consecutively). Conclusions SCUES showed similar validity and reliability with FMA-UE and SCUES can be used in the evaluation of upper extremity selective motor control in patients with stroke.

Measuring Ultrasonographic Thickness of the Achilles Tendon Insertion Is Less Reliable Than the Midportion in Healthy Tendons and Patients With Tendinopathy.

Ultrasound is the preferred imaging method in the diagnostic process of Achilles tendinopathy (AT). Ultrasound tissue characterization (UTC) is a frequently used, standardized and valid method to assess tendon geometry in AT patients. It is unknown whether UTC is reliable for measuring Achilles tendon thickness. The aim of the study was to assess intra- and inter-rater reliability of Achilles tendon thickness measurements using UTC in both asymptomatic individuals and patients with AT, and to evaluate if the reliability of thickness measurements differs between the midportion and insertional area. Exactly 50 patients with AT and 50 asymptomatic individuals were included. Using the conventional US and standardized UTC procedure maximum thickness was measured in the midportion and insertion region. To determine inter- and intra-rater reliabilities, the intraclass correlation coefficient (ICC) was used. The ICC values for inter- and intra-rater reliability were classified as "excellent," for the AT group (0.93 [95% CI: 0.88-0.96] and 0.95 [0.92-0.97]) and asymptomatic participants (0.91 [0.87-0.94] and 0.94 [0.92-0.96]). The reliability of measuring tendon thickness in the midportion region was "excellent," with both inter-rater (0.97 [0.95-0.98]) and intra-rater (0.98 [0.96-0.99]) ICC values indicating high levels of agreement. In the insertional region, ICC values for inter-rater (0.79 [0.69-0.87]) and intra-rater (0.89 [0.84-0.93]) reliability were "moderate to good." We showed excellent reliability for measuring the US thickness of the midportion and good reliability of measuring the insertional region in patients with AT. Significantly lower ICCs were observed for the reliability of thickness measurements in the insertional region when compared with the midportion.

Clinimetrics of the Lanarkshire Oximetry Index for patients with leg ulcers: A systematic review and meta-analysis.

Ankle Brachial Pressure Index (ABPI) measurement has long been considered the gold standard of vascular assessment for people with lower limb ulceration. Despite this, only around 15% of patients in the United Kingdom who require an ABPI measurement undergo the assessment. The Lanarkshire Oximetry Index (LOI) is a cheaper and arguably more accessible approach to vascular assessment and was initially proposed as an alternative to the ABPI in 2000. No synthesis of evidence related to the LOI has been performed since its introduction into the literature. Primary studies were sought to determine the clinimetric properties of the LOI and its level of agreement with ABPI assessments. Systematic searches of MEDLINE, CINAHL, Cochrane Central Register of Controlled Trials, BNI, ProQuest Health and Medicine, Science Direct, Google Scholar and the British Library (online search) were conducted. Reference lists of identified studies were also reviewed to identify additional studies. Three primary studies met the inclusion criteria, reporting data from 307 patients and 584 limbs assessed using both the LOI and ABPI. All three studies reported fair to moderate kappa values for interrater reliability (κ = 0.290-0.747) and statistically significant positive correlation coefficients (r = 0.37, p < 0.001 in two studies) between the LOI and ABPI. The combined data from the three studies indicated a sensitivity of 52% (41.78-62.1, 95% confidence interval [CI]) and specificity of 96.08% (93.4-97.9, 95% CI) for the LOI using the ABPI as a reference. Additional data are required to indicate the safety of the LOI in practice. Data are also required to determine if the LOI is more acceptable to clinicians compared to the ABPI and whether there are any barriers/enablers to its implementation in practice. Given the relatively low specificity of the LOI, it may be beneficial to combine measurement of the LOI with a subjective clinical risk assessment tool to improve the sensitivity of this alternative approach to vascular assessment.

Ultrasound measurement of subsynovial connective tissue thickness in the carpal tunnel: An intrarater/interrater reliability and agreement study

ObjectivesNon-inflammatory thickening of the subsynovial connective tissue (SSCT) in the carpal tunnel is commonly found in subjects with carpal tunnel syndrome (CTS), and quantification may shed light on CTS pathogenesis. To date, information on the reliability of ultrasound quantification of SSCT is scarce. Therefore, we investigated intrarater and interrater reliability/agreement for ultrasound quantification of SSCT thickness in subjects with and without CTS, and predictors for tissue thickness. Material and methodsTwo investigators quantified SSCT thickness and thickness ratio on ultrasound in 16 healthy subjects (age, 24–65 years; 16 left/14 right wrists) and 17 subjects with CTS (age, 37–83 years; 14 left/14 right wrists). Intra- and inter-rater reliability/agreement were assessed on intraclass correlation coefficients, standard error of measurement and minimal detectable change. A mixed-effects model was used to evaluate potential predictors for SSCT thickness. ResultsIntra- and inter-rater reliability analysis showed good to excellent intraclass correlation coefficients in both groups, ranging from 0.772 to 0.965. The maximum percentage standard error of measurement was 8%. The maximum minimal detectable change was 14% within raters, and 20% between raters. Both intra- and inter-rater reliability values for thickness ratio were poor. Presence of CTS (ß = 0.180; p = 0.015) correlated positively with SSCT thickness. ConclusionsUltrasound is a reliable method for quantification of SSCT thickness, but not for thickness ratio. Presence of CTS correlates positively with SSCT thickness.

Trust Your Fingers: Determining the Reliability and Validity of the Measurement of the Diastasis of the Rectus Abdominis Muscle by Palpation and Tape Measure

Background: Diastasis of the rectus abdominis muscle (DRAM) occurs in about 60% of pregnant women and often persists after delivery. Currently, the reference standard to measure DRAM is ultrasonography, and physiotherapy is considered first-line treatment. Most physiotherapists do not base their DRAM assessment on ultrasonography but instead on palpatory evaluation with tape measures. This method is neither standardized nor has been tested for its reliability or validity in women with interrectus distance (IRD) of 30 mm and greater. Objectives: To assess the reliability of the measurements of DRAM by palpation with tape measure and to evaluate its validity compared with ultrasonography. Study Design: Prospective observational study. Methods: Two physiotherapists assessed test-retest, intra- and interrater reliability of this method in 20 women with IRD of 30 mm and greater, using intraclass correlations (ICC) and compared the values with ultrasonography to establish its validity using Spearman ρ and Bland-Altman plots. The interval between the 2 sessions was 7 days and greater. Results: The ICC values for test-retest reliability ranged from 0.67 to 0.95 and between 0.40 and 0.85 for intrarater reliability. The ICC values for interrater reliability ranged from 0.13 to 0.29. In terms of validity, correlations were inconsistent with Spearman ρ ranging from −0.24 to 0.74. Conclusion: Single palpatory measurement with tape above the umbilicus and during partial curl-up might detect changes in IRD of 15.5 mm and greater over time in women with IRD of 30 mm and greater. It cannot replace ultrasonographic measurements in accuracy but might be the only available option for IRDs larger than the ultrasonographic probe.

Validity and Reliability of the Persian Version of Hypospadias Objective Penile Evaluation (HOPE) Questionnaire

Background and Objective: Surgical correction of hypospadias is one of the most common pediatric surgeries. Due to the several methods of hypospadias operations, the evaluation of results has always been challenging. Several questionnaires have been designed to reduce the discrepancy in the reports of hypospadias surgery, and the HOPE questionnaire is one of the most common. Based on our search, the reliability and validity of the Persian version of the HOPE (Hypospadias Objective Penile Evaluation) questionnaire have not been assessed yet. Materials and Methods: In this cross-sectional study, firstly, the questionnaire "Objective evaluation of penis in hypospadias" was translated by two urologists and an English language expert familiar with medical concepts according to the Backward-Forward translation method into Persian and then again into English, and finally, the questionnaire was sent to experts and compared with the original questionnaire. The quality of translation and content were finalized and corrected. Thereafter, in order to check the form and content validity, the questionnaire was sent to eight experts, and the content validity indices and its content validity ratio were calculated. In order to check the reliability-stability of the re-test, two questionnaires, including images after hypospadias surgery of 42 patients, were delivered to eight specialists in pediatrics, urology, and pediatric surgery to determine the score at a time interval of one week, and finally, the data were analyzed in SPSS software (version 23) using the test-retest method. The Intraclass Correlation Coefficient (ICC) index and its confidence interval were calculated by the scores obtained from twice answering the instrument for each factor and the entire questionnaire. In order to determine internal consistency and reliability, Cronbach's alpha coefficient was calculated for each factor and the entire questionnaire. Results: In our study, intraclass correlation coefficient values for inter-rater reliability in the HOPE score were calculated at 0.89, indicating the high reliability of this questionnaire in Persian. The internal consistency of the Persian version of the HOPE scale was confirmed, rendering a Cronbach's alpha coefficient of 0.98. Conclusion: The good validity and reliability of the HOPE scoring system in Persian support its use as a standard objective outcome measure of the cosmetic result after hypospadias surgery.

Reliability and agreement of the timed up and go test in children and teenagers with autism spectrum disorder

ASD patients include a variety of motor deficits; however, these issues have received less scientific attention than other ASD symptoms. Due to understanding and behavioral difficulties, it might be difficult to administer motor assessment measures to children and adolescents with ASD. To evaluate motor challenges in this population, including gait and dynamic balance issues, the timed up and go test (TUG) may be a simple, easy to apply, quick, and inexpensive tool. This test measures in seconds the time it takes for an individual to get up from a standard chair walk 3 m, turn around, walk back to the chair, and sit down again. The study purpose was to evaluate the inter- and intra-rater reliability of TUG test in children and teenagers with ASD. A total of 50 children and teenagers with ASD (43 boys and 7 girls) between 6 and 18 years were included. Reliability was verified by the intraclass correlation coefficient, standard error of measurement, and minimum detectable change. The agreement was analyzed by the Bland–Altman method. A good intra-rater reliability (ICC = 0.88; 95% CI = 0.79–0.93) and an excellent inter-rater reliability (ICC = 0,99; 95% CI = 0.98 to 0.99) were observed. Additionally, Bland–Altman plots demonstrated that there was no evidence of bias in either the replicates or between examiners. Furthermore, the limits of agreement (LOAs) between the testers and test replicates were close, indicating that there was little variation between measurements. Conclusions: The test TUG showed strong intra- and inter-rater reliability values, low proportion of measurement errors, and lack of significant bias based on by test repetition in children and teenagers with ASD. These results could be clinically useful for assessing balance and the risk of falls in children and teenagers with ASD. However, the present study is not free of limitations, such as the use of a non-probabilistic sampling.What is Known:• People with ASD have a variety of motor deficits that have a prevalence rate almost as common as intellectual disability. In our knowledge, there are no studies that provide data on the reliability of the use of scales or assessment tests in children and adolescents with ASD to measure motor difficulties, such as gait and dynamic balance, in children and teenagers with ASD.• Timed up and go test (TUG) could be a possible tool to measure this motor skills.What is New:• The reliability and agreement of the Timed up and go test in 50 children and teenagers with autism spectrum disorder showed strong intra- and inter-rater reliability values, low proportion of measurement errors, and lack of significant bias based on by test repetition.

A Comparative Evaluation of the Measurement Properties of Three Histological Indices of Mucosal Healing in Ulcerative Colitis: Geboes Score, Robarts Histopathology Index and Nancy Index.

To inform their future use in regulated clinical trials to evaluate treatment efficacy hypotheses, the measurement properties of three histological indices, Geboes Score [GS], Robarts Histopathology Index [RHI] and Nancy Index [NI], were evaluated among patients with ulcerative colitis. Analyses were conducted on data from a Phase 3 clinical trial of adalimumab [M14-033, n = 491] and focused on evaluating the measurement properties of the GS, RHI and NI. Specifically, internal consistency and inter-rater reliability, convergent, discriminant and known-group validity, and sensitivity to change were assessed at Baseline, and at Weeks 8 and 52. Internal consistency for the RHI showed lower alpha [α] values at Baseline [α = 0.62] relative to Weeks 8 [α = 0.82] and 52 [α = 0.81]. The inter-rater reliability values of RHI [0.91], NI [0.64] and GS [0.53] were excellent, good and fair, respectively. Regarding validity, Week 52 correlations were moderate to strong between full and partial Mayo scores and Mayo subscale scores and the RHI and GS, and were weak to moderate for the NI. Significant differences between mean scores of all three histological indices were observed across known-groups based on Mayo endoscopy subscores and full Mayo scores at Weeks 8 and 52 [p < 0.001]. The GS, RHI and NI are each capable of producing reliable and valid scores that are sensitive to changes in disease activity over time, in patients with moderately to severely active ulcerative colitis. While all three indices demonstrated relatively acceptable measurement properties, the GS and RHI performed better than the NI.

Unsatisfactory reproducibility of interstitial inflammation scoring in allograft kidney biopsy

Interstitial inflammation scoring is incorporated into the Banff Classification of Renal Allograft Pathology and is essential for the diagnosis of T-cell mediated rejection. However, its reproducibility, including inter-rater and intra-rater reliabilities, has not been carefully investigated. In this study, eight renal pathologists from different hospitals independently scored 45 kidney allograft biopsies with varying extents of interstitial inflammation. Inter-rater reliabilities and intra-rater reliabilities were investigated by kappa statistics and conditional agreement probabilities. Individual pathologists’ scoring patterns were examined by chi-squared tests and proportions tests. The mean pairwise kappa values for inter-rater reliability were 0.27, 0.30, and 0.26 for the Banff i score, ti score, and i-IFTA, respectively. No rater pair performed consistently better or worse than others on all three scorings. After dichotomizing the scores into two groups (none/mild and moderate/severe inflammation), the averaged conditional agreements ranged from 47.1% to 50.0%. The distributions of the scores differed, but some pathologists persistently scored higher or lower than others. Given the important role of interstitial inflammation scoring in the diagnosis of T-cell mediated rejection, transplant practitioners should be aware of the possible clinical implications of the far-from-optimal reproducibility.

Intrarater and interrater reliability of the dynamic gait index in post stroke patients with eye movement disorders

BackgroundThe Dynamic Gait Index (DGI) is a useful tool that has been evaluated for its reliability in patients with vestibular disorders, elderly people and, in chronic stroke population. Present study was aimed to evaluate the intrarater and interrater reliability of the DGI to measure dynamic balance and gait performance in stroke patients with eye movement disorders. MethodsA sample of 30 stroke patients suffering from eye movement disorders were recruited. Two Physical therapists assessed the intrarater and interrater reliability of the DGI in two testing sessions three days apart. In the later session, two raters assessed the patients’ performance simultaneously on the DGI. The reliability was calculated using the intra-class correlation coefficient (ICC2, 1). Standard error of measurement (SEM) and minimal detectable change (MDC95) at 95% confidence interval were also calculated. A significance level was set at p-value <0.05. ResultsThe (ICC2, 1) for intrarater and interrater reliability of total DGI scores was 0.86 and 0.91 respectively. While (ICC2, 1) for intrarater and interrater reliability of individual items ranged from 0.73 to 0.91 to 0.73–0.93, respectively. The (SEM) and (MDC95) for intrarater reliability of total DGI scores were 0.76 and 2.10, respectively. Corresponding values for interrater reliability were 0.62 and 1.71, respectively. ConclusionsThe DGI is a reliable tool for evaluating the dynamic balance and gait performance in stroke patients with eye movement disorders. This tool showed good to excellent intrarater and interrater reliability of total DGI scores and moderate to good intrarater and interrater reliability of individual items of the DGI.

Intraoperative arthroscopic classification tool for posterolateral elbow instability

Introducing and implementing an arthroscopic classification tool for posterolateral elbow instability. Thirty arthroscopies were performed on 30 patients, and all recordings were collected, blinded, and labeled. Three orthopedic surgeons reviewed and scored all 30 recordings three times with a period of at least seven days in between to analyze the intraobserver and interobserver reliability. The classification consisted of five different grades. Indications for elbow arthroscopy included impingement (n=7), osteochondritis dissecans (n=5), pain (n=7), osteoarthritis (n=6), and other (n=5). The kappa value for intrarater reliability was 0.71, indicating good reliability, while the kappa value for inter-rater reliability was 0.38 indicating fair reliability. This new classification is a tool for an arthroscopic assessment of PLRI and can be used as a standardized grading system for further research and communication between orthopedic surgeons. We demonstrated good intrarater reliability (k=0.71) with fair inter-rater reliability (k=0.38). However, further research is necessary to study the clinical significance.

Standardization and quality control of Doppler and fetal biometric ultrasound measurements in low-income setting.

The aim of this study was to determine the quality of fetal biometry and pulsed-wave Doppler ultrasound measurements in a prospective cohort study in Uganda. This was an ancillary study of the Ending Preventable Stillbirths by Improving Diagnosis of Babies at Risk (EPID) project, in which women enroled in early pregnancy underwent Doppler and fetal biometric assessment at 32-40 weeks of gestation. Sonographers undertook 6 weeks of training followed by onsite refresher training and audit exercises. A total of 125 images for each of the umbilical artery (UA), fetal middle cerebral artery (MCA), left and right uterine arteries (UtA), head circumference (HC), abdominal circumference (AC) and femur length (FL) were selected randomly from the EPID study database and evaluated independently by two experts in a blinded fashion using objective scoring criteria. Inter-rater agreement was assessed using modified Fleiss' kappa for nominal variables and systematic errors were explored using quantile-quantile (Q-Q) plots. For Doppler measurements, 96.8% of the UA images, 84.8% of the MCA images and 93.6% of the right UtA images were classified as of acceptable quality by both reviewers. For fetal biometry, 96.0% of the HC images, 96.0% of the AC images and 88.0% of the FL images were considered acceptable by both reviewers. The kappa values for inter-rater reliability of quality assessment were 0.94 (95% CI, 0.87-0.99) for the UA, 0.71 (95% CI, 0.58-0.82) for the MCA, 0.87 (95% CI, 0.78-0.95) for the right UtA, 0.94 (95% CI, 0.87-0.98) for the HC, 0.93 (95% CI, 0.87-0.98) for the AC and 0.78 (95% CI, 0.66-0.88) for the FL measurements. The Q-Q plots indicated no influence of systematic bias in the measurements. Training local healthcare providers to perform Doppler ultrasound, and implementing quality control systems and audits using objective scoring tools in clinical and research settings, is feasible in low- and middle-income countries. Although we did not assess the impact of in-service retraining offered to practitioners deviating from prescribed standards, such interventions should enhance the quality of ultrasound measurements and should be investigated in future studies. © 2022 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

Transcultural adaptation and assessment of psychometric properties of the Spanish version of the Scale for the Evaluation of Staff-Patient Interactions in Progress Notes.

To adapt the Scale for the Evaluation of Staff Patient Interactions in Progress Notes to Spanish and to test the psychometric properties. The study was conducted in two phases: (1) Adaptation of the instrument to Spanish following the Standards for Educational and Psychological Testing. (2) Psychometric study in a sample of mental health nurses. The Cronbach's alpha values were 0.97 for the total scale and 0.83 to 0.81 for each dimension. The inter-rater reliability values were between 0.94 and 0.97. The scale is a reliable tool for assessing nurses' clinical notes in relation to the quality of nurse-patient interactions.

The reliability of simultaneous versus individual data collection during stuttering assessment.

Speech-language pathologists often multitask in order to be efficient with their commonly large caseloads. In stuttering assessment, multitasking often involves collecting multiple measures simultaneously. The present study sought to determine reliability when collecting multiple measures simultaneously versus individually. Over two time periods, 50 graduate students viewed videos of four persons who stutter (PWS) and counted the number of stuttered syllables and total number of syllables uttered, and rated speech naturalness. Students were randomly assigned to one of two groups: the simultaneous group, in which all measures were gathered during one viewing; and the individual group, in which one measure was gathered per viewing. Relative and absolute intra- and inter-rater reliability values were calculated for each measure. The following results were notable: better intra-rater relative reliability for the number of stuttered syllables for the individual group (intraclass correlation coefficient (ICC) = 0.839) compared with the simultaneous group (ICC = 0.350), smaller intra-rater standard error of measurement (SEM) (i.e., better absolute reliability) for the number of stuttered syllables for the individual group (7.40) versus the simultaneous group (15.67), and better inter-rater absolute reliability for the total number of syllables for the individual group (88.29) compared with the simultaneous group (125.05). Absolute reliability was unacceptable for all measures across both groups. These findings show that judges are likely to be more reliable when identifying stuttered syllables in isolation than when simultaneously collecting them with total syllables spoken and naturalness data. Results are discussed in terms of narrowing the reliability gap between data collection methods for stuttered syllables, improving overall reliability of stuttering measurements, and a procedural change when implementing widely used stuttering assessment protocols. What is already known on the subject The reliability of stuttering judgments has been found to be unacceptable across a number of studies, including those examining the reliability of the most popular stuttering assessment tool, the Stuttering Severity Instrument (4th edition). The SSI-4, and other assessment applications, involve collecting multiple measures simultaneously. It has been suggested, but not examined, that collecting measures simultaneously, which occurs in the most popular stuttering assessment protocols, may result in substantially inferior reliability when compared to collecting measures individually. What this paper adds to existing knowledge The present study has multiple novel findings. First, relative and absolute intra-rater reliability were substantially better when stuttered syllables data were collected individually compared to when the same data were collected simultaneously with total number of syllables and speech naturalness data. Second, inter-rater absolute reliability for total number of syllables was also substantially better when collected individually. Third, intra-rater and inter-rater reliability were similar when speech naturalness ratings were given individually compared to when they were given while simultaneously counting stuttered and fluent syllables. What are the potential or actual clinical implications of this work? Clinicians can be more reliable when identifying stuttered syllables individually compared to when they judge stuttering along with other clinical measures of stuttering. In addition, when clinicians and researchers use current popular protocols for assessing stuttering that recommend simultaneous data collection, including the SSI-4, they should instead consider collecting stuttering event counts individually. This procedural change will lead to more reliable data and strongerclinical decision making.

Inter-rater reliability and prognostic value of baseline Radiographic Assessment of Lung Edema (RALE) scores in observational cohort studies of inpatients with COVID-19

ObjectivesTo reliably quantify the radiographic severity of COVID-19 pneumonia with the Radiographic Assessment of Lung Edema (RALE) score on clinical chest X-rays among inpatients and examine the prognostic value of...