Intraclass Correlation Coefficient Range Research Articles

The impact of skinfold thickness and exercise intensity on the reliability of NIRS in the vastus lateralis.

The aims of this study were (1) to assess the test-retest reliability of the primary near infrared spectroscopy (NIRS) variables (i.e., StO2, T[Hb], [HbO2] and [HHb]) during cycling and (2) to investigate potential influences of exercise intensity and adipose tissue thickness (ATT) on this reliability. 21 men and 20 women completed twelve constant work rate tests (6min) at six different exercise intensities with each intensity performed twice. NIRS variables were measured at the vastus lateralis. The coefficient of variance (CV%), the intraclass correlation coefficient (ICC), mean bias and limits of agreement (LoA) were determined for reliability purposes. The reliability of baseline values were acceptable to very good (CV% range: 5.83 - 21.96%). The reliability of end-values (CV% range: 0.02 - 25.02%, ICC range: 0.0 - 0.935) and amplitudes (CV% range: 0.46 - 5099%, ICC range: 0.0 - 0.887) were more variable. In general, the mean biases of end-values and amplitudes showed wide limits of agreement. A homogeneous influence of exercise intensity on reliability could not be established but reliability measures appeared to be lower in people with a lower skinfold thickness. Moreover, the NIRS signals decreased with increasing ATT but stabilized upon reaching a cut-off of 8mm ATT. In addition, ATT did have a significant influence on [HHb] amplitude. In participants with ATT < 8mm, higher amplitudes were observed with increasing intensity whereas in participants with ATT > 8mm, there were no differences between the intensities. The study reveals variable results with regards to reliability and there was no consistent influence of exercise intensity on reliability. Participants with a lower skinfold thickness showed stronger reliability. Moreover, NIRS signals decrease when ATT exceeds 8mm. Careful consideration is necessary when interpreting NIRS signals in such cases.

Quantitative measures of recent and lifetime agricultural pesticide use are associated with increased pesticide concentrations in house dust

ObjectiveElevated pesticide concentrations have been found in dust from homes with residents who use agricultural pesticides, but few studies have compared these concentrations to quantitative measures of their use. We evaluated household pesticide dust concentrations in relation to quantitative, active ingredient-specific metrics of agricultural pesticide use in the Biomarkers of Exposure and Effect in Agriculture Study. MethodsParticipants provided vacuum dust samples (2013–2018) and information regarding recent (last 12 months) and lifetime pesticide use. Thirty-two pesticide analytes were measured in 295 dust samples from 213 participants; 54 had repeated measurements (median = 96 days between visits). We used mixed-effects quantile regression models to estimate relative differences in pesticide concentrations for recent and lifetime agricultural use (number of days, intensity-weighted days), recent home/garden use (yes/no), and household characteristics. Only household characteristics were examined for dacthal because of no use information. We calculated intraclass correlation coefficients (ICCs) to evaluate temporal variability. We report only descriptive statistics for pesticides with detection rates <25 %. ResultsFor currently used pesticides, quantitative measures of recent agricultural use were associated with significantly increased household pesticide dust concentrations for malathion, metolachlor, acetochlor, cyfluthrin, and atrazine (p-trends < 0.001), but not permethrin. Similarly, quantitative measures of lifetime use were associated with increased concentrations of malathion, metolachlor, carbaryl, diazinon, and atrazine (p-trends < 0.001), but not permethrin, chlorpyrifos, or chlorothalonil. For banned pesticides, ever agricultural use was associated with elevated chlordane and heptachlor concentrations and non-significantly elevated dieldrin concentrations, but not lindane, p,p-DDD, p,p-DDE, or p,p-DDT. Recent home/garden use predicted increased malathion, carbaryl, and cyfluthrin concentrations. ICCs (range = 0.57–0.90) suggested moderate to high correlation over 3–6 months. Detection rates were <25 % for alachlor, butylate, EPTC, metribuzin, simazine, carbofuran, coumaphos, as well as for three banned pesticides (cyanazine, aldrin, endosulfan). ConclusionsHousehold pesticide dust concentrations were strongly associated with the frequency of agricultural pesticide use.

Reliability of the Star Excursion Balance Test with End-Stage Knee Osteoarthritis Patients and Its Responsiveness Following Total Knee Arthroplasty.

Background/Objectives: The Star Excursion Balance Test (SEBT) is a simple and feasible tool for assessing dynamic balance in individuals with knee osteoarthritis (KOA). It has an advantage as it replicates dynamic balance better than other static balance tools. This study aims to determine how reliable SEBT is among people with end-stage KOA, as well as how responsive it is and how well it correlates with performance-based outcome measures after TKA. Methods: Patients on the waiting list for TKA performed SEBT in the anterior, posteromedial and posteriorlateral directions twice within 7 days. The measurements were repeated 6 and 12 months after TKA. The participants completed performance-based outcome measurements (PBOMs) and the Oxford Knee Score (OKS) before and after TKA to estimate correlation. Results: In all directions, the intraclass correlation coefficient range (ICC) was 0.998-0.993, and there were no significant differences between the test and re-test mean SEBT scores. The standard error of measurement (SEM) ranged from 0.37% to 0.68%, and the minimum detectable change (MDC) ranged from 1.02% to 1.89%. The post TKA SEBT results show significant improvement, with a large effect size. There were large-to-medium correlations between SEBT and PBOMs before and after TKA, while OKS correlated only before surgery. The magnitude of change in SEBT, PBOMs and OKS did not correlate. Conclusions: SEBT is an extremely reliable tool for assessing dynamic balance in all three directions of severe KOA patients. It is sensitive enough to detect balance changes at 6 and 12 months post TKA. SEBT cannot be used to reflect the change in functional outcome improvement after TKA.

Food Records Show Daily Variation in Diet during Pregnancy: Results from the Temporal Research in Eating, Nutrition, and Diet during Pregnancy Study

BackgroundDiet is critical for pregnant individuals and their offspring, but insight into diet during pregnancy mainly comes from questionnaires and recalls. ObjectivesTo obtain detailed real-time dietary data during pregnancy to evaluate intra- and interindividual variation in intakes. MethodsPregnant individuals were recruited from a New York City health system December 2020–June 2023. Participants collected dietary intakes for 14 d (mean gestational weeks = 17.6) and again roughly 4 wk later (mean gestational weeks = 24.5). Participants logged each eating occasion using a smartphone and study-developed app, and wore a smartwatch capturing physiologic data. ResultsIn total, 150 individuals completed ≥1 data collection round, with 134 completing both rounds. Mean daily eating window was 10.82 h, with weekends having a significantly shorter window than weekdays (P < 0.001). Eating window was correlated with energy intake (r = 0.401, P < 0.001), driven by later last eating occasions. There was high intraindividual variation in macro- and micronutrient intakes [intraclass correlation coefficient (ICC), range 0.11–0.40] and food type (ICC range: 0.08–0.34), and differences between weekdays and weekends (less protein and micronutrients on weekends). Few participants’ mean intakes met daily recommended dietary allowances for key micronutrients (under 15% for iron, magnesium, vitamin D, and vitamin E; under 30% for calcium, folate, zinc, and vitamin A). ConclusionsDietary intakes varied substantially within and between individuals, and mean nutrient intake estimates did not capture nutrient adequacy for individuals or populations. Future work that examines individual daily dietary intakes throughout pregnancy among diverse populations is needed.

A feasibility study of magnetic resonance pelvimetry in women of short stature to identify the risk of cephalo-pelvic disproportion

ObjectiveTo conduct a feasibility study to assess if MR Pelvimetry can be undertaken in short stature primigravidae, with a view to a main study to answer the question ‘can MR pelvimetry be used to predict cephalo-pelvic disproportion (CPD) in women of short stature?’. Study DesignThis was a prospective single arm feasibility study. In the first phase, patient and public involvement was undertaken to assess acceptability of the study. In the second phase, primiparous women of ≤ 160 cm in height were selected from antenatal clinics. Obstetricians and midwives were asked to inform all eligible women about the study. Patients were approached in the third trimester and written information about the study provided. If the patient agreed to participate, they were invited for consent. MR scans were performed from 36 weeks gestation. Patients and their obstetricians were not informed of the results of the MR until after delivery (in view of NICE and WHO recommendations on pelvimetry). ResultsMR scans were performed on 21/35 (60%) participants who consented. The study was conducted without adverse events and was acceptable to all participants. Data were available from 19 patients; 7 (37%) of whom had caesarean section (CS) due to CPD, 7 (37%) had assisted vaginal birth (AVB), while 5 (26%) had spontaneous vaginal births (SVD). Two patients who had an elective CS were not included in the analysis. The pelvic measurements especially anatomical conjugate (inlet), transverse diameter (mid cavity) and anteroposterior diameter at the outlet were larger in the SVD group in comparison to in-labour CS /AVB. Interobserver MR scan measurements were comparable between radiologists: intraclass correlation coefficient (ICC) range 0.68 to 0.95. ConclusionThis feasibility study suggests that conducting a full study to answer the research question ‘can MR pelvimetry predict CPD in women of short stature?’would be feasible and acceptable to patients. The recruitment, MR scan protocol, reporting, reliability of measures were all assessed and found acceptable for a trial. A trend was observed for smaller pelvic measurements in women who needed intervention compared with those who had a natural birth/SVD. With evidence from a full trial, this could have major implications for the management of short stature women, in terms of clinical practice and safety. Key messagesA large trial would be feasible and acceptable to assess whether MR pelvimetry can predict CPD in women of short stature. Despite the small number of patients, a trend was observed for smaller pelvic measurements in women who needed intervention compared to those who had spontaneous vaginal birth. Evidence from a large trial could help inform clinical practice, and provide information and choice for women at risk of CPD.

Intersystem and Interoperator Agreement of US Attenuation Coefficient for Quantifying Liver Steatosis.

Background The extent of liver steatosis can be assessed using US attenuation coefficient (AC) algorithms currently implemented in several US systems. However, little is known about intersystem and interoperator variability in measurements. Purpose To assess intersystem and interoperator agreement in US AC measurements for fat quantification in individuals with varying degrees of liver steatosis and to assess the correlation of each manufacturer's AC algorithm results with MRI proton density fat fraction (PDFF). Materials and Methods This prospective study was conducted at Southwoods Imaging, Youngstown, Ohio, September 30-October 1, 2023. Two operators independently obtained AC measurements using eight US systems equipped with an AC algorithm from different manufacturers. On the same day, MRI PDFF measurement was performed by a different operator. Correlation between US AC and MRI PDFF was assessed using a mixed-effects model. Agreement between systems and operators was evaluated using the intraclass correlation coefficient (ICC). Results Twenty-six individuals (mean age, 55.4 years ± 10.7 [SD]; 16 female participants) were evaluated. The correlation of US AC with MRI PDFF was high for five AC algorithms (r range, 0.70-0.86), moderate for two (r = 0.62 for both), and poor for one (r = 0.47). In pairwise comparisons, none of the pairs of systems achieved excellent agreement (overall ICC = 0.33 [95% CI: 0.15, 0.52]). One pair showed good agreement (ICC = 0.79 [95% CI: 0.66, 0.87]), eight pairs showed moderate agreement (ICC range, 0.50 [95% CI: 0.22, 0.69] to 0.73 [95% CI: 0.49, 0.85]), and 19 pairs showed poor agreement (ICC range, 0.11 [95% CI: -0.06, 0.37] to 0.48 [95% CI: 0.20, 0.67]). Interoperator agreement on AC value was excellent for the Samsung Medison algorithm (ICC = 0.90 [95% CI: 0.80, 0.96]), good for the Siemens Healthineers (ICC = 0.76 [95% CI: 0.54, 0.89]) and Canon Medical Systems (ICC = 0.76 [95% CI: 0.16, 0.92]) algorithms, and moderate for the remaining algorithms (ICC range, 0.50 [95% CI: 0.16, 0.73] to 0.74 [95% CI: 0.51, 0.88]). The mean AC value obtained by the two operators did not differ for any system except the system from Canon Medical Systems. Conclusion There was substantial variability in AC values obtained with different US systems, precluding interchangeability between systems for liver steatosis diagnosis and follow-up imaging. Interoperator agreement ranged from moderate to excellent. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Han in this issue.

Evaluation of the reliability of measuring lower back muscles cross-sectional area based on manual segmentation within multi-level MRI images

IntroductionManual segmentation of paraspinal muscle cross-sectional area (CSA) is widely used to assess related health disorders. This study aimed to evaluate the reliability of this segmentation process for each paraspinal region of interest across the three intervertebral levels commonly used for segmentation (L3/4, L4/5, and L5/S1). MethodsAxial-T2-weighted MRI images for 238 patients were divided among five raters (47 ± 1 cases each). To conduct the intra-rater reliability study, the CSA of each paraspinal lumber muscle (psoas major (PM), multifidus (MF), and erector spinae (ES)) and the intervertebral disc (ID) were manually segmented twice on all targeted levels before being assessed for each region per rater. The Inter-rater reliability was determined by comparing the results of different readers who segmented the same dataset. The Intraclass Correlation (ICC) and Coefficient-of-Variation percentage (CV%) were reported for each analysis. ResultsLow intra- and inter-rater variability (CV%<11) was found for each reader in each region and intervertebral level. The inter-rater reliability was excellent (ICC>0.9) for the PM, ID, and ES at L3/4. However, it was very good for MF at all levels and ES at L4/5, L5/S1 (ICC range: 0.82–0.88) affected by the fat-infiltration nature of the ES and MF muscles, their proximity to each other, and their smaller size (correlation between muscle size and ICC = 0.6, P < 0.01). The ID segmentation has the lowest CV (<3 %) and excellent ICC (>0.93). ConclusionManual paraspinal muscle segmentation using axial-T2-weighted MRI is reliable at all commonly segmented intervertebral levels. However, the reliability level can be degraded by the presence of high-fat infiltrate, unclear muscle boundaries, and muscle size. Implications for practiceFollowing consistent guidelines can help improve segmentation results. The IDs can be used as reliable internal references.

Evaluation of Smartphone Technology on Spatiotemporal Gait in Older and Diseased Adult Populations.

Objective: Advancements in smartphone technology provide availability to evaluate movement in a more practical and feasible manner, improving clinicians' ability to diagnose and treat adults at risk for mobility loss. The purpose of this study was to evaluate the validity and reliability of a smartphone application to measure spatiotemporal outcomes during level (primary) and uphill/downhill (secondary) walking with and without an assistive device for older adults (OAs), Parkinson's Disease (PD) and cerebrovascular accident (CVA) populations. Methods: A total of 50 adults (OA = 20; PD = 15; CVA = 15) underwent gait analysis at self-selected gait speeds under 0-degree, 5-degree uphill and 5-degree downhill environments. The validity and reliability of the smartphone outcomes were compared to a motion-capture laboratory. Bland-Altman analysis was used to evaluate limits of agreement between the two systems. Intraclass correlation coefficients (ICCs) were used to determine absolute agreement, and Pearson correlation coefficients (r) were used to assess the strength of the association between the two systems. Results: For level walking, Bland-Altman analysis revealed relatively equal estimations of spatiotemporal outcomes between systems for OAs without an assistive device and slight to mild under- and overestimations of outcomes between systems for PD and CVA with and without an assistive device. Moderate to very high correlations between systems (without an assistive device: OA r-range, 0.72-0.99; PD r-range, 0.87-0.97; CVA r-range, 0.56-0.99; with an assistive device: PD r-range, 0.35-0.98; CVA r-range, 0.50-0.99) were also observed. Poor to excellent ICCs for reliability between systems (without an assistive device: OA ICC range, 0.71-0.99; PD ICC range, 0.73-0.97; CVA ICC range, 0.56-0.99; with an assistive device: PD ICC range, 0.22-0.98; CVA ICC range, 0.44-0.99) were observed across all outcomes. Conclusions: This smartphone application can be clinically useful in detecting most spatiotemporal outcomes in various walking environments for older and diseased adults at risk for mobility loss.

Reliability of a generative artificial intelligence tool for pediatric familial Mediterranean fever: insights from a multicentre expert survey

BackgroundArtificial intelligence (AI) has become a popular tool for clinical and research use in the medical field. The aim of this study was to evaluate the accuracy and reliability of a generative AI tool on pediatric familial Mediterranean fever (FMF).MethodsFifteen questions repeated thrice on pediatric FMF were prompted to the popular generative AI tool Microsoft Copilot with Chat-GPT 4.0. Nine pediatric rheumatology experts rated response accuracy with a blinded mechanism using a Likert-like scale with values from 1 to 5.ResultsMedian values for overall responses at the initial assessment ranged from 2.00 to 5.00. During the second assessment, median values spanned from 2.00 to 4.00, while for the third assessment, they ranged from 3.00 to 4.00. Intra-rater variability showed poor to moderate agreement (intraclass correlation coefficient range: -0.151 to 0.534). A diminishing level of agreement among experts over time was documented, as highlighted by Krippendorff’s alpha coefficient values, ranging from 0.136 (at the first response) to 0.132 (at the second response) to 0.089 (at the third response). Lastly, experts displayed varying levels of trust in AI pre- and post-survey.ConclusionsAI has promising implications in pediatric rheumatology, including early diagnosis and management optimization, but challenges persist due to uncertain information reliability and the lack of expert validation. Our survey revealed considerable inaccuracies and incompleteness in AI-generated responses regarding FMF, with poor intra- and extra-rater reliability. Human validation remains crucial in managing AI-generated medical information.

Comparison of PD-L1 assays in head and neck carcinoma

Programmed cell death-ligand 1 (PD-L1) expression is a predictive biomarker for response to immune checkpoint inhibitor in head and neck squamous cell carcinoma. Given the range of antibodies and platforms for PD-L1 testing, it is essential to understand the performance of different staining and scoring methods. PD-L1 expression in 156 head and neck mucosal squamous cell carcinoma (HNmSCC) cases at Asan Medical Center was assessed using 106 tissue microarray (TMA) cores and 50 whole slides. Three standardised PD-L1 assays (22C3 pharmDx, SP263, and 28-8 pharmDx) and one laboratory-developed test (22C3 LDT) were evaluated: the combined positive score (CPS) with ≥1, ≥20, and ≥50 cut-offs, and the tumour positive score (TPS) with ≥1%, ≥20%, ≥50% cut-offs. Concordance on a continuous scale among the assays was good to excellent for CPS [intraclass correlation coefficient (ICC) range 0.73-0.94] and TPS (ICC range 0.70-0.94) and in both TMA and whole slides cohorts. Stratification by variable cut-offs demonstrated moderate to good agreement among most assays, as analysed by Gwet's AC1. PD-L1 expression was significantly correlated with tumour location using the 22C3 pharmDx assay (CPS, p=0.014; TPS, p=0.033). Notable concordance was found among PD-L1 assays, suggesting their potential interchangeability in HNmSCC.

Assessing the performance of AI-assisted technicians in liver segmentation, Couinaud division, and lesion detection: a pilot study

BackgroundIn patients with primary and secondary liver cancer, the number and sizes of lesions, their locations within the Couinaud segments, and the volume and health status of the future liver remnant are key for informing treatment planning. Currently this is performed manually, generally by trained radiologists, who are seeing an inexorable growth in their workload. Integrating artificial intelligence (AI) and non-radiologist personnel into the workflow potentially addresses the increasing workload without sacrificing accuracy. This study evaluated the accuracy of non-radiologist technicians in liver cancer imaging compared with radiologists, both assisted by AI.MethodsNon-contrast T1-weighted MRI data from 18 colorectal liver metastasis patients were analyzed using an AI-enabled decision support tool that enables non-radiology trained technicians to perform key liver measurements. Three non-radiologist, experienced operators and three radiologists performed whole liver segmentation, Couinaud segment segmentation, and the detection and measurements of lesions aided by AI-generated delineations. Agreement between radiologists and non-radiologists was assessed using the intraclass correlation coefficient (ICC). Two additional radiologists adjudicated any lesion detection discrepancies.ResultsWhole liver volume showed high levels of agreement between the non-radiologist and radiologist groups (ICC = 0.99). The Couinaud segment volumetry ICC range was 0.77–0.96. Both groups identified the same 41 lesions. As well, the non-radiologist group identified seven more structures which were also confirmed as lesions by the adjudicators. Lesion diameter categorization agreement was 90%, Couinaud localization 91.9%. Within-group variability was comparable for lesion measurements.ConclusionWith AI assistance, non-radiologist experienced operators showed good agreement with radiologists for quantifying whole liver volume, Couinaud segment volume, and the detection and measurement of lesions in patients with known liver cancer. This AI-assisted non-radiologist approach has potential to reduce the stress on radiologists without compromising accuracy.Graphical abstract

Perseverance reduces whereas impulsivity increases the risk of reoffending

By law, every criminal verdict describes the norm violation in question. To date, the use of this information for prognostic purposes has been standardized only for sex offenses. This study presents a content-analytic approach to the material. A theory-based coding system allows for systematic tracing of the offender’s affective and regulatory processing during norm violation, targeting the entire action sequence: initial activation affect (positive vs. negative), impulsivity vs. intentionality, affect regulation, and coping with intention-implementation difficulties. In a sample of N = 304 males convicted of violent and/or sexual offenses, the processing variables were investigated with respect to (a) objective measurability, (b) relationships with established reoffending risk markers, and (c) their (incremental) performance in predicting reoffending. With respect to objectivity (a), the inter-rater agreement (n = 111) varied widely among the variables (intraclass correlation range 0.03–0.62). Regarding risk markers (b), significant associations were found with the Level of Service Inventory–Revised (LSI-R) subscales criminal history, alcohol/drugs, emotional/personal, and attitudes/orientation. The Crime Scene Behavior Risk Measure for sexual offenses (n = 106) correlated with psychological processing variables for the items multiple juvenile offenders and male victim. Two variables emerged as relevant for predicting recidivism (c): negatively motivated impulsivity predicted violent reoffending, while persistence when coping with implementation difficulties negatively predicted sexual and general reoffending. These relationships statistically mediated the predictive contribution of actuarial risk tools (LSI-R/crime scene behavior risk). Psychological processes during norm violation as reported in verdicts appear to represent underexploited information within risk assessment and beyond.

Reproducibility of Assessment of Lepidic (Noninvasive) Patterns in Lung Adenocarcinoma With Cytokeratin Immunostain Compared With Hematoxylin and Eosin and the Proposed New International Association for the Study of Lung Cancer (IASLC) Algorithm

IntroductionLepidic growth is considered noninvasive in lung nonmucinous adenocarcinoma, whereas other patterns are invasive. Considerable interobserver variability in assessing “invasion” has been reported. We assessed the utility of cytokeratin 7 (CK7) stain and recently proposed International Association for the Study of Lung Cancer criteria to improve assessment of noninvasion in lung adenocarcinoma. MethodsFour pathologists (two staff, two trainees) assessed 158 hematoxylin and eosin (HE)- and CK7-stained slides of 108 pT1N0-2 nonmucinous lung adenocarcinoma cases. Scoring took place in four rounds. First, sections were independently scored for percentage of noninvasive or probable noninvasive and invasive or probable invasive patterns. Second, after a consensus scoring algorithm for CK7 was formulated, the slides were rescored. Subsequent third-round scoring was conducted only on HE slides using the 2023 International Association for the Study of Lung Cancer proposed criteria, and fourth-round scoring on both HE and CK7 slides simultaneously. Intraclass correlation coefficient (ICC) was calculated for each round. Recurrence-free survival was assessed using Cox proportional hazards regression methods. ResultsIn the first two rounds, interobserver concordance was consistently higher with CK7 (ICC range = 0.44–0.6) than HE (range = 0.24–0.49) scores. The IASLC proposed algorithm improved ICC of HE scores to 0.60 (95% confidence interval: 0.52–0.67), and round 4 HE and CK7 combined improved ICC to 0.75 (95% confidence interval: 0.70–0.80). Continuous measures of averaged noninvasive and probable noninvasive scores on HE were associated with improved recurrence-free survival (hazard ratio: 0.83–0.86). ConclusionsCK7 staining consistently increased interobserver concordance in assessment of invasive versus noninvasive patterns than HE. Combining CK7 with the 2023 IASLC criteria for morphologic features of invasion may further improve the interobservers’ concordance for the recognition of lepidic growth in nonmucinous lung adenocarcinoma.

How Reliable Is the Measurement of the Lateral Center Edge Angle on Scoliosis Radiographs for Detecting Acetabular Dysplasia?

Hip dysplasia reportedly occurs in up to 12% of the general population and may be higher in patients with adolescent idiopathic scoliosis (AIS). When using lateral center edge angle (LCEA) measurements to help identify hip dysplasia, it is uncertain if this measurement can be performed reliably on scoliosis radiographs due to parallax. The purpose of this study is to evaluate the reliability of measuring LCEA on scoliosis radiographs compared with standard pelvis radiographs. This study evaluated 40 hips on 20 patients (mean age 12.5 years ± 3.1; 50% Female) who received PA scoliosis radiographs and AP pelvis radiographs within 1 year of each other. Review was performed by 4 pediatric radiologists (2 general, 2 MSK), 2 pediatric orthopaedic surgeons, and 1 research fellow. Radiographic measurement of the LCEA for each hip was first performed on the scoliosis radiograph. After a minimum of 3 days, the LCEA was measured on a pelvis radiograph of the same individual obtained within 1 year of the scoliosis radiograph. Pearson coefficient was used to measure agreement between scoliosis and pelvis radiograph measurements. Intraclass correlation coefficient (ICC) was used to evaluate intraobserver and interobserver agreement. ICC values <0.5 were classified as poor reliability, 0.5 to 0.75 were classified moderate, 0.75 to 0.90 were classified good, and >0.9 indicated excellent reliability. The mean difference between scoliosis and pelvis radiographs was 54 ± 79 days. There was good-to-excellent interobserver agreement on LCEA measurements made between readers on scoliosis radiographs (ICC: 0.94, 95% CI: 0.90-0.96, P <0.001) and pelvis radiographs (ICC: 0.91, 95% CI: 0.83-0.95, P <0.001), and moderate-to-excellent intraobserver agreement for scoliosis radiographs (ICC range: 0.68 to 0.98; P <0.001) and pelvis radiographs (ICC range: 0.62 to 0.96; P <0.001). There was a strong correlation between LCEA measurements made on scoliosis and pelvis radiographs ( r2 =0.66, P <0.001), and the intermodality agreement between scoliosis and pelvis radiograph LCEA measurements were moderate to good (ICC range: 0.68 to 0.89, P <0.001). Overall, there was good-to-excellent agreement between readers on scoliosis and pelvis radiographs, respectively, and moderate-to-excellent intraobserver agreement between LCEA measurements made on scoliosis radiographs and pelvis radiographs, respectively. LCEA measurements made on scoliosis radiographs strongly correlated to the measurements made on pelvis radiographs, and the intermodality ICC was also considered moderate to good. Dedicated pelvis radiographs may not be necessary during scoliosis workup and follow-up surveillance, thereby decreasing radiation exposure, cost, and improving patient care workflow. Level IV-diagnostic study.

Reproducibility of rCBV in glioblastomas using T2*-weighted perfusion MRI: an evaluation of sampling, normalization, and experience.

The reproducibility of relative cerebral blood volume (rCBV) measurements among readers with different levels of experience is a concern. This study aimed to investigate the inter-reader reproducibility of rCBV measurement of glioblastomas using the hotspot method in dynamic susceptibility contrast perfusion magnetic resonance imaging (DSC-MRI) with various strategies. In this institutional review board-approved single-center study, 30 patients with glioblastoma were retrospectively evaluated with DSC-MRI at a 3.0 Tesla scanner. Three groups of reviewers, including neuroradiologists, general radiologists, and radiology residents, calculated the rCBV based on the number of regions of interest (ROIs) and reference areas. For statistical analysis of feature reproducibility, the intraclass correlation coefficient (ICC) and Bland-Altman plots were used. Analyses were made among individuals, reader groups, reader-group pooling, and a population that contained all of them. For individuals, the highest inter-reader reproducibility was observed between neuroradiologists [ICC: 0.527; 95% confidence interval (CI): 0.21-0.74] and between residents (ICC: 0.513; 95% CI: 0.20-0.73). There was poor reproducibility in the analyses of individuals with different levels of experience (ICC range: 0.296-0.335) and in reader-wise and group-wise pooling (ICC range: 0.296-0.335 and 0.397-0.427, respectively). However, an increase in ICC values was observed when five ROIs were used. In an analysis of all strategies, the ICC for the centrum semiovale was significantly higher than that for contralateral white matter (P < 0.001). The inter-reader reproducibility of rCBV measurement was poor to moderate regardless of whether it was calculated by neuroradiologists, general radiologists, or residents, which may indicate the need for automated methods. Choosing five ROIs and using the centrum semiovale as a reference area may increase reliability for all users.

The Bethesda ERCP Skills Assessment Tool (BESAT) can reliably differentiate endoscopists of different experience levels.

Background and study aims The Bethesda ERCP Skill Assessment Tool (BESAT) is a video-based assessment tool of technical endoscopic retrograde cholangiopancreatography (ERCP) skill with previously established validity evidence. We aimed to assess the discriminative validity of the BESAT in differentiating ERCP skill levels. Methods Twelve experienced ERCP practitioners from tertiary academic centers were asked to blindly rate 43 ERCP videos using the BESAT. ERCP videos consisted of native biliary cannulation and sphincterotomy and were recorded from 10 unique endoscopists of various ERCP experience (from advanced endoscopy fellow to > 10 years of ERCP experience). Inter-rater reliability, discriminative validity, and internal structure validity were subsequently assessed. Results The BESAT was found to reliably differentiate between endoscopists of varying levels of ERCP experience with experienced ERCPists scoring higher than novice ERCPists in 11 of 13 (85%) instrument items. Inter-rater reliability for BESAT items ranged from good to excellent (intraclass correlation range: 0.86 to 0.93). Internal structure validity was assessed with item-total correlations ranging from 0.53 to 0.83. Conclusions Study findings demonstrate that the BESAT, a video-based ERCP skill assessment tool, has high inter-rater reliability and has discriminative validity in differentiating novice from expert ERCP skill. Further investigations are needed to determine the role of video-based assessment in improving trainee learning curves and patient outcomes.

Validation of a fully automatic three-dimensional assessment of orthognathic surgery

The aim of the present study was to propose and validate FAST3D: a fully automatic three-dimensional (3D) assessment of the surgical accuracy and the long-term skeletal stability of orthognathic surgery. To validate FAST3D, the agreement between FAST3D and a validated state-of-the-art semi-automatic method was calculated by intra-class correlation coefficients (ICC) at a 95 % confidence interval. A one-sided hypothesis test was performed to evaluate whether the absolute discrepancy between the measurements produced by the two methods was statistically significantly below a clinically relevant error margin of 0.5 mm. Ten subjects (six male, four female; mean age 24.4 years), class II and III, who underwent a combined three-piece Le Fort I osteotomy, bilateral sagittal split osteotomy and genioplasty, were included in the validation study. The agreement between the two methods was excellent for all measurements, ICC range (0.85–1.00), and fair for the rotational stability of the chin, ICC = 0.54. The absolute discrepancy for all measurements was statistically significantly lower than the clinical relevant error margin (p < 0.008). Within the limitations of the present validation study, FAST3D demonstrated to be reliable and may be adopted whenever appropriate in order to reduce the work load of the medical staff.

Cross-informant agreement between caregivers and teachers for prosocial behavior across child welfare settings

Healthy social functioning has been a growing focus for understanding children’s positive development and well-being. Despite the flexibility and cost-effectiveness of behavior rating scales, there has been a rising issue of concern in both practice and research applications regarding rating scales and its low cross-informant agreement. The present study aimed to analyze the extent to which caregivers and teachers agree about children’s prosocial behaviors, identify whether there were differences in the degree of agreement across child welfare settings, and ultimately offer recommendations for assessing behavior for children involved with child welfare services. To do so, the reports of 1224 children, caregivers, and teachers from the National Survey of Child and Adolescent Well-Being (NSCAW) I who have been investigated by Child Protective Services (CPS) for abuse and/or neglect have been examined. Intra-class correlation coefficients (ICCs) of caregiver and teacher assessment of children’s prosocial functioning were calculated to understand the degree of agreement in their perceptions of children’s prosocial behaviors. ICC values presented show that agreement was low but significant across all samples and subscales (ICC range, 0.11–0.22). It is suggested that future researchers pursue examination of the underlying mechanisms or factors that contribute to informant discrepancies.

The validity and reliability of the Urostomy Education Scale: a methodological study.

There are no scales used by stoma and wound care nurses to evaluate the stoma care skills of individuals with a urostomy. This study was conducted to investigate the Turkish validity and reliability of the Urostomy Education Scale (UES). The study sample consisted of 66 patients who had undergone radical cystectomy, were in the 0- to 7-day postoperative period, were older than 18 years of age, had no physical or mental disabilities, had no urostomy complications, and agreed to participate in the study. This study used translation and back-translation to determine the linguistic validity of the UES in Türkiye. Expert opinion was consulted for content validity. Then, 2 competent and 2 experienced nurses evaluated the face validity of the scale with 5 individuals with urostomies. Reliability of the scale was assessed using internal consistency, interrater reliability, and intraclass correlation coefficients. The content validity index was 0.81 and α = 0.66 to 0.95 for the Cronbach's alpha of the competent nurse assessment and α = 0.68 to 0.96 in the expert nurse assessment. The intraclass correlation coefficient (ICC) results indicated sufficient and statistically significant agreement (ICC range: 0.6-1) between the evaluations made by the 2 evaluators for each skill. The Turkish version of the 7-item UES is a valid and reliable tool that can be used to determine the self-care levels of individuals with a urostomy.

A Video-Based Procedure-Specific Competency Assessment Tool for Minimally Invasive Esophagectomy

Minimally invasive esophagectomy (MIE) is a complex procedure with substantial learning curves. In other complex minimally invasive procedures, suboptimal surgical performance has convincingly been associated with less favorable patient outcomes as assessed by peer review of the surgical procedure. To develop and validate a procedure-specific competency assessment tool (CAT) for MIE. In this international quality improvement study, a procedure-specific MIE-CAT was developed and validated. The MIE-CAT contains 8 procedural phases, and 4 quality components per phase are scored with a Likert scale ranging from 1 to 4. For evaluation of the MIE-CAT, intraoperative MIE videos performed by a single surgical team in the Esophageal Center East Netherlands were peer reviewed by 18 independent international MIE experts (with more than 120 MIEs performed). Each video was assessed by 2 or 3 blinded experts to evaluate feasibility, content validity, reliability, and construct validity. MIE-CAT version 2 was composed with refined content aimed at improving interrater reliability. A total of 32 full-length MIE videos from patients who underwent MIE between 2011 and 2020 were analyzed. Data were analyzed from January 2021 to January 2023. Performance assessment of transthoracic MIE with an intrathoracic anastomosis. Feasibility, content validity, interrater and intrarater reliability, and construct validity, including correlations with both experience of the surgical team and clinical parameters, of the developed MIE-CAT. Experts found the MIE-CAT easy to understand and easy to use to grade surgical performance. The MIE-CAT demonstrated good intrarater reliability (range of intraclass correlation coefficients [ICCs], 0.807 [95% CI, 0.656 to 0.892] for quality component score to 0.898 [95% CI, 0.846 to 0.932] for phase score). Interrater reliability was moderate (range of ICCs, 0.536 [95% CI, -0.220 to 0.994] for total MIE-CAT score to 0.705 [95% CI, 0.473 to 0.846] for quality component score), and most discrepancies originated in the lymphadenectomy phases. Hypothesis testing for construct validity showed more than 75% of hypotheses correct: MIE-CAT performance scores correlated with experience of the surgical team (r = 0.288 to 0.622), blood loss (r = -0.034 to -0.545), operative time (r = -0.309 to -0.611), intraoperative complications (r = -0.052 to -0.319), and severe postoperative complications (r = -0.207 to -0.395). MIE-CAT version 2 increased usability. Interrater reliability improved but remained moderate (range of ICCs, 0.666 to 0.743), and most discrepancies between raters remained in the lymphadenectomy phases. The MIE-CAT was developed and its feasibility, content validity, reliability, and construct validity were demonstrated. By providing insight into surgical performance of MIE, the MIE-CAT might be used for clinical, training, and research purposes.