Intraclass Correlation Coefficient Estimates Research Articles

Approaches for Quantifying the ICC in Multilevel Logistic Models: A Didactic Demonstration

Multilevel modeling techniques have gained traction among experimental psychologists for their ability to account for dependencies in nested data structures. Increasingly, these techniques are extended to the analysis of binary data (e.g., correct or incorrect responses). Despite their popularity, the information in logistic multilevel models is often underutilized when researchers focus solely on fixed effects and ignore important heterogeneity that exists between participants. In this tutorial, we review four techniques for estimating and quantifying the relative degree of between-person variability in logistic multilevel models in an accessible manner using real data. First, we introduce logistic multilevel modeling, including the interpretation of fixed and random effects. Second, we review the challenges associated with the estimation and interpretation of within- and between-participant variation in logistic multilevel models, particularly computing the intraclass correlation coefficient (ICC), which is usually a first, simple step in a linear MLM. Third, we demonstrate four existing methods of quantifying the ICC in logistic multilevel models and discuss their relative advantages and disadvantages. Fourth, we present bootstrapping methods to make statistical inference about these ICC estimates. To facilitate reuse, we developed R code to implement the discussed techniques, which is provided throughout the text and as supplemental materials.

KovaaK's aim trainer as a reliable metrics platform for assessing shooting proficiency in esports players: a pilot study.

Esports research lacks game-based metrics platforms appropriate for adequately capturing esports performance. The aim of this pilot study was to assess the reliability of the KovaaK's first-person shooter (FPS) aim trainer as a metrics platform for assessing shooting proficiency in esports players. Ten FPS esports players completed two identical experimental trials (T) separated by three to five days. Each trial included four rounds (R) of testing, evaluating four shooting tasks: Micro Flicking, Macro Flicking, Strafe Tracking, and Wall Peeking. Reliability of performance outcomes (e.g., accuracy, headshot accuracy, hits per second, and total shots hit) were assessed using the intraclass correlation coefficient (ICC) and their 95% confidence intervals (CI), and significant differences were identified using repeated-measures analysis of variance (RM-ANOVA). Results indicated excellent, or good to excellent reliability for all outcome variables with the ICC estimates ranging between 0.947-0.995, with lower and upper bound 95% CIs ranging between 0.876-0.988, and 0.984-0.999, respectively. Significant improvements were seen between experimental trials in the Macro Flicking task for accuracy (p = .005) and hits per second (p = .009) only. Significant interactions between trial and round were identified in the Micro Flicking task for accuracy (p = .006), with post hoc analysis showing accuracy was significantly higher in T1R1 compared to T2R1 (87.74 ± 3.13 vs. 85.99 ± 3.05, respectively, p = .02), and in T2R4 compared to T2R2 (87.99 ± 2.89 vs. 84.70 ± 4.25, respectively, p = .049). Significant interactions were also identified in the Strafe Tracking task for headshot accuracy (p = .002), with post hoc analysis showing headshot accuracy was significantly higher in T1R2 compared to T2R2 (78.48 ± 8.15 vs. 76.79 ± 12.16, respectively, p = .003), and in T1R2 compared to T1R1 (78.48 ± 8.15 vs. 73.68 ± 17.94, respectively, p = .023). In summary, this study demonstrates that KovaaK's provides a reliable metrics platform for assessing shooting proficiency in esports, however, some variability in performance was observed.

Concurrent validity and test-retest reliability of Squegg™ - smart dynamometer and handgrip trainer in healthy individuals.

Handgrip strength is crucial for overall health and is often evaluated using hand dynamometers. Two hand dynamometers were used in the study: the Jamar® Hydraulic Hand Dynamometer and Squegg™ Smart Dynamometer and Handgrip Trainer, which is a relatively new, Bluetooth-enabled device to measure handgrip strength. The purpose of the study was to compare Jamar and Squegg to determine concurrent validity and, separately, test-retest reliability of Squegg. The cross-sectional study measured bimanual handgrip strength using the Jamar and Squegg in two sessions, following standard protocols. Participants were 18+ years old, with no hand disease or dysfunction. Spearman's rank correlations were used to evaluate concurrent validity between Jamar and Squegg. Test-retest reliability was evaluated with intraclass correlation coefficient (ICC) estimates and 95% confidence intervals calculated using a mean-rating (k = 2) absolute agreement, 2-way mixed effects model. Data from 594 participants were used to evaluate concurrent validity. Moderately strong correlations between Jamar and Squegg values for the right hand (RH) (ρ = 0.670; p = .001) and Jamar and Squegg values for the left hand (LH) (ρ = 0.730; p = .001) were found, indicating 'good' concurrent validity. Test-retest reliability for Squegg ranged from good to excellent (RH, ICC = 0.911, 95% CI (0.895, 0.924), p = .001; LH, ICC = 0.928, 95% CI (0.915, 0.939), p = .001). The Squegg demonstrated good concurrent validity with Jamar and good-to-excellent test-retest reliability. It can be recommended as an alternative to the Jamar as a handgrip strength measurement tool. Squegg, which uses Bluetooth technology, is a reliable and cost-effective option for consistent handgrip strength testing.

Investigating Equivalence of In-Person and Telehealth-Based Neuropsychological Assessment Performance for Individuals Being Investigated for Younger Onset Dementia.

Given the rapid shift to in-home teleneuropsychology models, more research is needed to investigate the equivalence of non-facilitator models of teleneuropsychology delivery for people with younger onset dementia (YOD). This study aimed to determine whether equivalent performances were observed on neuropsychological measures administered in-person and via teleneuropsychology in a sample of people being investigated for YOD. Using a randomized counterbalanced cross-over design, 43 participants (Mage = 60.26, SDage = 7.19) with a possible or probable YOD diagnosis completed 14 neuropsychological tests in-person and via teleneuropsychology, with a 2-week interval. Repeated measures t-tests, intraclass correlation coefficients (ICC), and Bland Altman analyses were used to investigate equivalence across the administration conditions. No statistical differences were found between in-person and teleneuropsychology conditions, except for the Hospital Anxiety and Depression Scale Anxiety subtest. Small to negligible effect sizes were observed (ranging from .01 to .20). ICC estimates ranged from .71 to .97 across the neuropsychological measures. Bland Altman analyses revealed that the Wechsler Adult Intelligence Scale-Fourth Edition Block Design subtest had slightly better overall performance in the in-person condition and participants reported higher levels of anxiety symptoms during the teleneuropsychology condition; however, average anxiety symptoms remained within the clinically normal range. Participants reported a high level of acceptability for teleneuropsychology assessments. These results suggest that performances are comparable between in-person and teleneuropsychology assessment modalities. Our findings support teleneuropsychology as a feasible alternative to in-person neuropsychological services for people under investigation of YOD, who face significant barriers in accessing timely diagnoses and treatment options.

Estimating individual health-related quality of life changes in low back pain patients

BackgroundThere is a need to evaluate different options for estimating individual change in health-related quality of life for patients with low back pain.MethodsSecondary analysis of data collected at baseline and 6 weeks later in a randomized trial of 749 adults with low back pain receiving usual medical care (UMC) or UMC plus chiropractic care at a small hospital at a military training site or two large military medical centers. The mean age was 31; 76% were male and 67% were White. The study participants completed the Patient-Reported Outcomes Measurement Information System (PROMIS®)-29 v 1.0 physical function, pain interference, pain intensity, fatigue, sleep disturbance, depression, anxiety, satisfaction with participation in social roles, physical summary, and mental health summary scores (T-scored with mean = 50 and standard deviation (SD) = 10 in the U.S. general population).ResultsReliability estimates at the baseline ranged from 0.700 to 0.969. Six-week test–retest intraclass correlation estimates were substantially lower than these estimates: the median test–retest intraclass correlation for the two-way mixed-effects model was 0. 532. Restricting the test–retest reliability estimates to the subset who reported they were about the same as at baseline on a retrospective rating of change item increased the median test–retest reliability to 0.686. The amount of individual change that was statistically significant varied by how reliability was estimated, and which SD was used. The smallest change needed was found when internal consistency reliability and the SD at baseline were used. When these values were used, the amount of change needed to be statistically significant (p < .05) at the individual level ranged from 3.33 (mental health summary scale) to 12.30 (pain intensity item) T-score points.ConclusionsWe recommend that in research studies estimates of the magnitude of individual change needed for statistical significance be provided for multiple reliability and standard deviation estimates. Whenever possible, patients should be classified based on whether they 1) improved significantly and perceived they got better, 2) improved significantly but did not perceive they were better, 3) did not improve significantly but felt they got better, or 4) did not improve significantly or report getting better.

Evaluation of laser power stability of repeatedly used SubCyclo probe in micropulse transscleral cyclophotocoagulation for glaucoma: A step towards sustainable ophthalmic surgery

Purpose To evaluate the laser power stability of the SubCyclo probe for micropulse transscleral cyclophotocoagulation after repeated use. Materials and methods This experimental study involved 6 new probes. Each probe was connected to the SubCyclo mode (2,000 mW power, 31.3% duty cycle, and 100 seconds duration) of the Vitra 810 laser delivery system (Quantel Medical, France). Laser power measurements were taken using a calibrated laser power meter (Nova, Ophir Optronics Solutions, Israel) every 10 seconds from 10 to 90 seconds during each of the 40 cycles. Intra-rater reliability was assessed using intraclass correlation (ICC). A linear mixed model for repeated measures and pairwise comparisons with Bonferroni adjustment were used for the analysis. Results The mean (SD) power outputs of all probes for the first cycle and all cycles were 421.9 (19.7) mW and 436.7 (16.1) mW, respectively. During the first cycle, the mean (SD) laser power gradually decreased from 444.3 (13.4) mW at 10 seconds to 407.3 (17.0) mW at 90 seconds (Fig 3). For all cycles, the power was 446.0 (13.6) mW at 10 seconds and gradually declined to 426.8 (21.0) mW at 90 seconds. Pairwise comparisons revealed significant differences in mean laser power outputs after 16 cycles of repeated use compared to the first cycle. The ICC estimate (95% CI) for intra-rater reliability was 0.96 (0.89, 0.99). Conclusions The SubCyclo probe maintains stable laser power outputs throughout repeated use for up to 16 cycles, with a significant increase observed after 16 cycles.

The level of agreement between the numerical rating scale and visual analogue scale for assessing pain intensity in adults with chronic pain.

The numerical rating scale and visual analogue scale are used to quantify pain intensity. However, it has not yet been explored whether these scores are interchangeable in adults with chronic pain. Data from the prospective multicentre cross-sectional INTERVAL study were used to evaluate the one-dimensionality and agreement between numerical rating scale scores and visual analogue scale scores in adults with chronic pain. Pain intensity scores using the numerical rating scale and visual analogue scale were provided by 366 patients with chronic pain for current, average, minimal and maximal pain. To evaluate whether pain intensity scales are completed in accordance with each other, the proportion of patients who satisfied the following condition was calculated: minimal pain intensity ≤ maximal pain intensity. A factor analysis confirmed the one-dimensionality of the pain measures. A significant difference was found between numerical rating scale and visual analogue scale scores for average, current, minimum and maximum pain. Intra-class correlation coefficient estimates ranged from 0.739 to 0.858 and all measures failed to show sufficient and acceptable agreement at the 95% level. The strength of agreement between pain severity categories was classified as 'moderate' for average and minimal pain and 'substantial' for current and maximal pain. The proportion of patients who scored minimal pain ≤ maximal pain was 97.5% for the numerical rating scale and 89.5% for the visual analogue scale. This study failed to show an acceptable agreement between the numerical rating scale and visual analogue scale when pain intensity was rated by adults with chronic pain, despite showing both scales measure the same information.

Within-individual variation of measured depression symptoms: A systematic review and meta-analysis

BackgroundDepression screening instruments are commonly used to assess the presence and severity of depression symptoms. However, there is little information on variation in depression screening scores over time within the same individual. MethodsA systematic review and meta-analysis was performed of studies reporting the within-subject variability of the Beck's Depression Inventory (BDI), Hospital Anxiety and Depression Scale (HADS), Patient Health Questionnaire-9 (PHQ-9) and Patient Health Questionnaire-2 (PHQ-2). Multiple databases were searched from inception to 14th July 2023. Title and abstract screening was performed in duplicate, full text screening and data extraction by one reviewer and verified by a second. Risk of bias was assessed with a modified COSMIN tool. ResultsOf 2798 titles and abstracts and 157 full text articles screened, 41 met the inclusion criteria. No studies were on patients with depression, most had only two measurements, less than two weeks apart. The pooled estimates of ICCs (Intraclass Correlations) and 95 % confidence intervals for BDI, HADS, PHQ-9 and PHQ-2 were 0.89 (0.84–0.93), 0.89 (0.85–0.93), 0.84 (0.81–0.88) and 0.75 (0.56–0.93), respectively.Studies in healthy subgroups showed lower ICC than those with physical illness. LimitationsAssessment of variability is not the main aim of most papers assessing depression measurement instruments, so it is possible that some relevant papers have been missed. ConclusionsWithin-subject scores for BDI, HADS and PHQ-9, show generally excellent agreement over periods of up to two weeks. However published data on within-subject variability is lacking over longer time periods and for patients with depression.

Alternative Endoscopy Reading Paradigms Determine Score Reliability and Effect Size in Ulcerative Colitis.

Central reading of endoscopy is advocated by regulatory agencies for clinical trials in ulcerative colitis [UC]. It is uncertain whether the local/site reader should be included in the reading paradigm. We explore whether using locally- and centrally-determined endoscopic Mayo subscores [eMS] provide a reliable final assessment and whether the paradigm used has an impact on effect size. eMS data from the TURANDOT [NCT01620255] study were used to retrospectively examine seven different reading paradigms (using the scores of local readers [LR], first central readers [CR1], second central readers [CR2], and various consensus reads [ConCR]) by assessing inter-rater reliabilities and their impact on the key study endpoint, endoscopic improvement. More than 40% of eMS scores between two trained central readers were discordant. Central readers had wide variability in scorings at baseline (intraclass correlation coefficient [ICC] of 0.475 [0.339, 0.610] for CR1 vs CR2). Centrally-read scores had variable concordance with LR (LR vs CR1 ICC 0.682 [0.575, 0.788], and LR vs CR2 ICC 0.526 [0.399, 0.653]). Reading paradigms with LR and CR which included a consensus, enhanced ICC estimates to >0.8. At Week 12, without the consensus reads, the CR1 vs CR2 ICC estimates were 0.775 [0.710, 0.841], and with consensus reads the ICC estimates were >0.9. Consensus-based approaches were most favourable to detect a treatment difference. The ICC between the eMS of two trained and experienced central readers is unexpectedly low, which reinforces that currently used central reading processes are still associated with several weaknesses.

The role of sound touch elastography in assessment of liver fibrosis in chronic liver disease keeping APRI as the reference standard.

The study aims to determine the role of Sound Touch Elastography [STE] technique in staging liver fibrosis and predicting clinically significant gastro-esophageal varices among patients with chronic liver disease [CLD] keeping aspartate aminotransferase to platelet ratio index [APRI] as the reference standard. A prospective short-term study including 60 eligible patients with CLD were staged as non-significant fibrosis [NSF], significant fibrosis [SF] and cirrhosis [C] based on APRI values. STE was performed on each patient obtaining multiple readings as per pre-defined standards. The intra-observer reliability between each measurement and its association with APRI staging was evaluated using relevant statistical variables. Further, Youden's index was used to define the optimum cut-off values on STE in differentiating the stages of fibrosis and in predicting clinically significant gastro-esophageal varices. Based on APRI cut-off values, 41.7% [n=25] of the study population had cirrhosis, while 45% [n=27] had significant fibrosis and 13.3% [n=8] had NSF. The STE values in kPa showed a positive correlation with APRI values [(rs)=0.837, p<0.001]. The intra-class correlation estimates based on a mean rating [k=5] was found to be 0.97 [0.95-0.99], implying an excellent agreement between the measurements. Optimum cut-off values in staging SF and C were 7.26 kPa [J=0.73, sensitivity-85.19%, specificity-87.5%; 95% CI] and 13.79 kPa [J=0.84, sensitivity-96.0%, specificity-88.89%; 95% CI]. The AUROC for each of these stages were 0.926 [0.785-0.987] and 0.976 [0.890-0.999], respectively. 23.3% [n=14] of the study population had clinically significant gastro-esophageal varices with a value above 18.84 kPa [J=0.88] showing a sensitivity of 92.85% and a specificity of 95.65% in predicting the same. The novel STE technique shows good accuracy in staging liver fibrosis as determined by APRI values and in prediction of clinically significant gastro-esophageal varices with excellent reliability. It shows promising prospects and can be integrated widely in clinical practice for assessment and staging of fibrosis in CLD.

An Application of Theril Indexes for the Interrater Reliability: A Comparison with Intraclass Correlations

This study proposes to apply the Theil-index ratios for the interrater reliability. We discuss the theoretical foundations and examine its function empirically using real data. Our analyses show that Theil-index rations and intraclass correlation (ICC) estimates are highly correlated. However, ICC may underestimate the interrater reliability by some extreme disagreement among raters and be more likely to be influenced by the extreme disagreement. As Theil-index ratios overcome the limitations of ICC to some degree, it seems that Theil-index ratios provide an alternative to evaluating interrater reliability, at least under certain conditions, e.g., when outliers exist in the data, it is difficult to obtain the variance component estimates, or ICC underestimates interrater reliability.

Exploring the usefulness of the Brief COPE in clinical and positive psychology: A discriminant content validity study

The Brief Coping Orientation to Problem Experienced (COPE) scale is widely used for measuring coping strategies. However, concerns persist regarding the dimensions captured by the scale and the accuracy of item representation. This study examined the relevance of adapted Brief COPE items using discriminant content validity (DCV). A panel of experts (n = 15) assessed the extent to which the items corresponded to their intended dimensions. Intraclass correlation (ICC) estimates ranged from 0.640 to 0.828, indicating agreement among experts. A one-sample t-test evaluated DCV, revealing that 21 out of 28 items distinctly and exclusively measured intended dimensions, confirming their discriminant content validity. Seven items were excluded: three did not measure coping strategies (non-dimension), and four measured them in different dimensions (wrong-dimension). The discriminant content-validated Brief COPE scale improves coping assessment, benefiting psychological therapies and providing researchers with refined measures for each coping strategy dimension, addressing dimensional concerns.

Reliability of the Modified Method for Sphericity Deviation Score Using Only the Involved Hip Radiographs in the Patients With Legg-Calve-Perthes Disease.

The sphericity deviation score (SDS) is a validated radiographic outcome measure that quantifies the severity of the femoral head deformity in the healed stage of Legg-Calve-Perthes disease (LCPD). The current method requires radiographs of both hips regardless of unilateral involvement to standardize radiographic magnification. As LCPD affects the unilateral hip in 85% to 90% of cases, the current method imposes unnecessary radiation exposure to most of the patients and having to exclude patients from research studies who only had unilateral hip radiographs. We thus modified the SDS method to use unilateral hip radiographs. The purpose of this study was to investigate the reliability of the modified SDS method using the radiographs containing only one hip. This retrospective study involved 40 patients with LCPD with unilateral involvement in the healed stage of LCPD. We modified the SDS measurement method by using the distance between the teardrop and the lateral acetabulum for magnification correction and providing a clear anatomic description of reference points on the femoral head. Three independent observers performed measurements using radiographs containing the affected hip only (modified method) and both hips (conventional method). The intraclass correlation (ICC) estimates were calculated. To verify clinical relevance, the correlation of the SDS with the Stulberg classification and hip range of motion (ROM) were also investigated. Measurements using the modified SDS showed excellent inter-/intra-observer ICCs (0.903 to 0.978). The ICCs between the modified and conventional methods were also excellent: 0.940 to 0.966 within the same observer and 0.897 to 0.919 between different observers. The modified SDS showed moderate-to-strong correlations with the Stulberg classification (Spearman ρ=0.650) and hip ROM (Pearson r =-0.661). The modified SDS measurement method showed excellent inter-/intra-observer reliability and moderate-to-strong correlations with the Stulberg classification and hip ROM. This method will help reduce unnecessary radiation exposure in patients with unilateral LCPD and prevent the exclusion of patients with unilateral radiographs from future research studies. Level III-diagnostic study.

A systematic evaluation of normalization methods and probe replicability using infinium EPIC methylation data

BackgroundThe Infinium EPIC array measures the methylation status of > 850,000 CpG sites. The EPIC BeadChip uses a two-array design: Infinium Type I and Type II probes. These probe types exhibit different technical characteristics which may confound analyses. Numerous normalization and pre-processing methods have been developed to reduce probe type bias as well as other issues such as background and dye bias.MethodsThis study evaluates the performance of various normalization methods using 16 replicated samples and three metrics: absolute beta-value difference, overlap of non-replicated CpGs between replicate pairs, and effect on beta-value distributions. Additionally, we carried out Pearson’s correlation and intraclass correlation coefficient (ICC) analyses using both raw and SeSAMe 2 normalized data.ResultsThe method we define as SeSAMe 2, which consists of the application of the regular SeSAMe pipeline with an additional round of QC, pOOBAH masking, was found to be the best performing normalization method, while quantile-based methods were found to be the worst performing methods. Whole-array Pearson’s correlations were found to be high. However, in agreement with previous studies, a substantial proportion of the probes on the EPIC array showed poor reproducibility (ICC < 0.50). The majority of poor performing probes have beta values close to either 0 or 1, and relatively low standard deviations. These results suggest that probe reliability is largely the result of limited biological variation rather than technical measurement variation. Importantly, normalizing the data with SeSAMe 2 dramatically improved ICC estimates, with the proportion of probes with ICC values > 0.50 increasing from 45.18% (raw data) to 61.35% (SeSAMe 2).

Measuring the Quality of Statutory Social Workers’ Communication Skills with Young People: Reliability of the My Social Work Partner Scales

Abstract Statutory meetings are at the core of social work practice, yet there is no quantitatively tested common set of criteria for assessing statutory social workers’ communication skills with young people. Reliably measuring such skills is important for the training and supervision of social workers as well for drawing conclusions about whether such skills impact service users’ outcomes. To fill this gap, we developed a new measure, the My Social Work Partner (MRP in Danish abbreviation) scales, which measure eleven dimensions of statutory communication. We tested the inter-coder reliability and intra-coder reliability of codings on the MRP scales using the intra-class correlation coefficient (ICC) in a random sample of thirty videotapes of statutory meetings. Five professional social workers participated in the inter-coder reliability test; three of them coded the same films again after a three- to five-month interval for the intra-coder reliability test. ICC estimates of inter-coder reliability ranged from good to excellent. Intra-coder reliability estimates ranged from fair to excellent. Overall, the results are promising and support the use of the MRP scales in social work practice with young persons, education and research. Future research will focus on how to improve the intra-coder reliability of the scales, in particular the Dialogue structure scale.

INTERRATER RELIABILITY OF AN ADVANCED NONCONTACT WOUND MEASUREMENT SYSTEM USING ARTIFICIAL INTELLIGENCE

Abstract Venous leg ulcers (VLU) are recurring, disabling wounds that develop primarily in older adults with comorbidities. In an ongoing clinical trial testing a nutritional intervention researchers use advanced technology (Swift Skin and Wound) to measure VLU healing over time. This noncontact wound measurement system uses artificial intelligence to automatically calculate wound measurements and track healing progress. However, a subjective component of the Swift system involves delineating wound perimeters on photographs captured with a tablet or smartphone. To evaluate interrater reliability of the system in the current study, measurements of 11 wounds by two independent raters were assessed using an intraclass correlation coefficient (ICC). ICC estimates and their 95% confidence intervals were calculated using SPSS statistical package version 27 based on a mean-rating (k = 2), consistency, 2-way mixed-effects model. For area measures, ICC = 0.99 with 95% confidence interval = 0.998-1.0, indicating the Swift measurement system has excellent interrater reliability.

Reliability of the ISAKOS Modification to Subclassify Rockwood Type III Acromioclavicular Joint Injuries.

To improve clinical management, the International Society of Arthroscopy, Knee Surgery and Orthopaedic Sports Medicine (ISAKOS) developed a specific subclassification of Rockwood type III acromioclavicular (AC) injuries: IIIA (no overriding distal clavicle) and IIIB (overriding distal clavicle). The study aimed to determine the inter- and intraobserver reliability of the radiographic classification proposed by ISAKOS for AC injuries. It was hypothesized that the strength of agreement for the ISAKOS modification will be substantial to almost perfect. Cohort study (diagnosis); Level of evidence, 3. We evaluated 40 radiographs of all types of AC joint injuries from patients at a single institution. The images were distributed to 6 shoulder and elbow fellowship-trained orthopaedic surgeons, along with standardized assessment questionnaires. The evaluators measured the bilateral coracoclavicular distance and the coracoclavicular ratio and classified the severity of the injuries according to the modified ISAKOS Rockwood classification. Four of the surgeons repeated the evaluation 6 weeks later to calculate intraobserver agreement. The kappa (κ) statistic was calculated for categorical inter- and intraobserver reliability. Intraclass correlation coefficient (ICC) estimates were calculated for the reliability of the coracoclavicular distance measurement. A 2-tailed test was performed to assess statistical significance. Overall interobserver agreement was substantial (κ = 0.637; 95% CI, 0.595-0.680) among the 6 evaluators, and intraobserver agreement was also substantial among the 4 evaluators (κ = 0.616; 95% CI, 0.549-0.638). The interobserver agreement for evaluating types IIIA and IIIB was fair (κ = 0.215; 95% CI, 0.135-0.295) and moderate (κ = 0.473; 95% CI, 0.393-0.553), respectively. The agreement on the coracoclavicular distance measurements was excellent among the 6 evaluators on both the affected side (ICC, 0.982; 95% CI, 0.970-0.990) and the unaffected side (ICC, 0.930; 95% CI, 0.894-0.958). Substantial agreement was found when categorizing AC joint injuries using the ISAKOS modification of the Rockwood classification, with excellent reliability demonstrated for the quantitative assessment of vertical displacement of the clavicle. Visual examination was unreliable in differentiating type IIIA injuries from type IIIB injuries.

Sample size recommendations for studies on reliability and measurement error: an online application based on simulation studies

Simulation studies were performed to investigate for which conditions of sample size of patients (n) and number of repeated measurements (k) (e.g., raters) the optimal (i.e., balance between precise and efficient) estimations of intraclass correlation coefficients (ICCs) and standard error of measurements (SEMs) can be achieved. Subsequently, we developed an online application that shows the implications for decisions about sample sizes in reliability studies. We simulated scores for repeated measurements of patients, based on different conditions of n, k, the correlation between scores on repeated measurements (r), the variance between patients’ test scores (v), and the presence of systematic differences within k. The performance of the reliability parameters (based on one-way and two-way effects models) was determined by the calculation of bias, mean squared error (MSE), and coverage and width of the confidence intervals (CI). We showed that the gain in precision (i.e., largest change in MSE) of the ICC and SEM parameters diminishes at larger values of n or k. Next, we showed that the correlation and the presence of systematic differences have most influence on the MSE values, the coverage and the CI width. This influence differed between the models. As measurements can be expensive and burdensome for patients and professionals, we recommend to use an efficient design, in terms of the sample size and number of repeated measurements to come to precise ICC and SEM estimates. Utilizing the results, a user-friendly online application is developed to decide upon the optimal design, as ‘one size fits all’ doesn’t hold.

Use and repeatability of 3D light scanning to measure transverse dorsal profile size and symmetry in the thoracic region in horses

Equine epaxial muscle size, thoracolumbar profile and symmetry in horses is of clinical interest due to relationships with pain and pathology. Flexible-curve rulers have previously been used to gather reliable, objective measures of thoracic profile, however, 3D light-scanning offers a potential non-contact alternative method to estimate cross sectional area (CSA) of the region. 3D light-scans of the thoracic epaxial region were taken from ten endurance horses (7 geldings, 3 mares; 8±2 years). Total CSA of the combined epaxial musculature, using computer software, was calculated at scapula and T18 levels (depth: 15 cm). Intra and inter-rater (n=3) reliability of CSA measurements was assessed using Friedman’s analyses and post-hoc Wilcoxon rank tests (three repeated measures). Intraclass correlation estimates (ICC ± 95% confidence intervals (CI)) were calculated (mean-rating, absolute-agreement, 2-way mixedeffects model). Paired t-tests assessed differences between right and left areas. No significant differences existed for transverse plane-cuts (scapula, T18 P&gt;0.05) between light-scans. Right and left areas were significantly different at the withers (P=0.012) with the left side larger in 70% of scans, but no significant differences were found between sides at T18. No differences existed for different plane-cuts of the same horse (P=0.53; ICC: 0.76; CIs: 0.43-0.92). While reliability was reduced between all raters (P=0.02; ICC: 0.70; CIs: 0.56-0.82), no significant differences occurred between two different assessors experienced in using the software (P=0.88; ICC: 0.90; CIs: 0.82-0.95). Intra-rater reliability for assessing thoracic profile and inter-rater reliability ICC values with experienced analysts was interpreted as good/excellent. The results suggest 3D light-scanning is an objective, non-invasive method to record size and symmetry of the epaxial region in horses and warrants validity testing against current measurement methods such as the flexible-curve ruler.

Non-Expert Markings of Active Chronic Graft-Versus-Host Disease Photographs: Optimal Metrics of Training Effects.

Lack of reliable measures of cutaneous chronic graft-versus-host disease (cGVHD) remains a significant challenge. Non-expert assistance in marking photographs of active disease could aid the development of automated segmentation algorithms, but validated metrics to evaluate training effects are lacking. We studied absolute and relative error of marked body surface area (BSA), redness, and the Dice index as potential metrics of non-expert improvement. Three non-experts underwent an extensive training program led by a board-certified dermatologist to mark cGVHD in photographs. At the end of the 4-month training, the dermatologist confirmed that each trainee had learned to accurately mark cGVHD. The trainees' inter- and intra-rater intraclass correlation coefficient estimates were "substantial" to "almost perfect" for both BSA and total redness. For fifteen 3D photos of patients with cGVHD, the trainees' median absolute (relative) BSA error compared to expert marking dropped from 20 cm2 (29%) pre-training to 14 cm2 (24%) post-training. Total redness error decreased from 122 a*·cm2 (26%) to 95 a*·cm2 (21%). By contrast, median Dice index did not reflect improvement (0.76 to 0.75). Both absolute and relative BSA and redness errors similarly and stably reflected improvements from this training program, which the Dice index failed to capture.