Agreement Intraclass Correlation Research Articles

Stability of Blood DNA Methylation Across Two Timepoints in Three Cohorts

Background: DNA methylation mediates the gene–environment interactions, with implications for health and disease. Studies with sampling at more than one timepoint revealed the considerable variability of the blood methylome, but comprehensive resources on genome-wide methylation stability are still lacking. We aimed to identify methylation sites that remain the most stable across two timepoints in human whole blood. Methods: Publicly available blood DNA methylation data from three cohorts were analysed, which included methylation profiles at two timepoints >1 year apart. The cohorts included pre-/post-pubertal children (Illumina 450k array), the elderly (Illumina 450k array), and middle-aged adults with obesity (Illumina EPIC array). Two metrics were used for the stability assessment: the mean absolute difference (MAD) of beta values between two measurements and the intraclass correlation coefficient (ICC). We searched for probes demonstrating high stability (low MAD and high ICC) across the three cohorts. Data from 51 children, 86 elderly adults, and 120 middle-aged participants were re-analysed. Results: The median interquartile range (IQR) of the maximum (from three datasets) MAD was 2.1% (1.5–2.9%), and the median of the minimum ICC agreement coefficient was 0.053 (−0.077–0.304). The Pearson’s correlation coefficient for the ICC vs. maximum MAD was low (r = 0.34, p < 2.2 × 10−16). We found only 239 probes that were highly stable based on both the maximum MAD (<5th percentile, <0.01) and ICC criterion (>95th percentile, >0.74). Conclusions: The whole-blood DNA methylation profile, as measured using microarrays, is dynamic over >1 year, but contains a fraction of stable probes, most of which are related to genomic variation. A resource describing probe stability is made publicly available, with the intention to support biomarker studies and the investigation of early epigenetic programming. The absolute error and correlation are two complementary facets of probe stability that may be considered in further research, especially to determine the stability of probes in health and disease across different tissues and populations.

Inter-reader reliability and diagnostic accuracy of PI-RADS scoring between academic and community care networks: How wide is the gap?

ImportanceThe Prostate Imaging Reporting & Data System (PI-RADS) scoring guidelines were developed to address the substantial variation in interpretation and reporting of prostate cancer (PCa) multiparametric MRI (mpMRI) results, and subsequent updates have sought to further improve inter-reader reliability. Nonetheless, the variability of PI-RADS scoring in real-world settings may represent a continuing challenge to the widespread standardization of prostate mpMRI and limit its overall clinical benefit. ObjectiveTo assess variability in mpMRI interpretation and reporting of PCa, we evaluated the discrepancies in PI-RADS scoring between community practices and a tertiary academic care center. Design, setting, and participantsWe identified 262 mpMRI studies from nonacademic facilities, reinterpreted by radiologists at our institution between January 2016 and July 2022. Results of targeted MRI fusion biopsy were identified for 193 of these patients, totaling 302 lesions. PI-RADS scoring from both community and academic interpreters were recorded in addition to presence of clinically significant PCa (csPCa) on pathological analysis of targeted cores. Main Outcome and measuresThe primary outcome was inter-reader reliability via intraclass correlation (ICC) and the kappa statistic. We also assessed the diagnostic accuracy of PI-RADS scoring for detecting csPCa for both cohorts via receiver operator characteristics (ROC) analysis and compared these findings using paired-sample area difference under curve analysis. ResultsInter-reader agreement and reliability of PI-RADS scoring per lesion was generally poor (absolute agreement ICC = 0.393, 95% CI: 0.288–0.488; consistency ICC = 0.407, 95% CI: 0.308–0.497; kappa = 0.336, 95% CI: 0.267–0.406). Reliability results from studies obtained after the publication of PI-RADSv2.1 were similar to those of the overall analysis. No agreement was observed in the subgroup of lesions scored as PIRADS 3 by community interpreters. No statistically significant difference in diagnostic accuracy was observed between cohorts (ROC area under curve [AUC]: 0.759 vs. 0.785, respectively; P = 0.337). PI-RADS 3 was determined to be the optimal cutoff for detecting clinically significant disease in both cohorts. Conclusions and relevanceOur results suggest that mpMRI diagnostic accuracy for detecting csPCa is not significantly different between academic and community practices. However, significantly poor reliability of mpMRI was observed between cohorts, suggesting the risk of introducing practice variation for community prostate cancer management. Variability, particularly for PI-RADS 3 lesions, can lead to inconsistent biopsy recommendations, which may result in missed csPCa or unnecessary biopsies.

Asymptotic Confidence Interval, Sample Size Formulas and Comparison Test for the Agreement Intra-Class Correlation Coefficient in Inter-Rater Reliability Studies.

The agreement intra-class correlation coefficient (ICCa) is a suitable statistical index for inter-rater reliability studies. With balanced Gaussian data, we prove the explicit form of ICCa asymptotic normality (ASN), valid both with analysis of variance (ANOVA), maximum likelihood (ML), or restricted ML (REML) estimates. An asymptotic confidence interval is then derived and its performances are examined by simulation compared to the most commonly used methods, under small, moderate and large sample size designs. Then, we deduce sample size calculation formulas, for the number of subjects and observers needed, to achieve a desired confidence interval width or an acceptable ICCa value test power and give concrete examples of their use. Finally, we propose a likelihood ratio test (LRT) to compare two ICCa's from two distinct subpopulations of patients (or raters) and study by simulation its first order risk and power properties. These methods are illustrated using data from two inter-rater reliability studies, one in physiotherapy with 42 patients and 10 raters and the second in neonatology with 80 subjects and 14 raters. In conclusion, we made recommendations to employ the proposed confidence interval for medium to large samples combined with the quantification of the minimal required sample size at the planning step, or the posterior-power at the analysis step, using simple dedicated formulas. Furthermore, with sufficient sizes, the proposed LRT seems suitable to compare inter-rater reliability between two patient subpopulations. Used wisely, this proposed methods toolbox can remedy common current issues in inter-rater reliability studies.

Development and testing of a tool to assess parental care for children with mental illness (PCCMI).

Involving parents in mental health treatment services can empower parents to manage their child's illness. We conducted this study to develop a tool to assess parental care for children with mental illness. The present study was conducted in five phases that included in-depth interviews with parents in the outpatient department (OPD), focus group discussions (FGDs) with parents in an in-patient setting, and with a multidisciplinary team, item development, and the assessment of psychometric properties of the tool. Parental care for children with mental illness (PCCMI) has 50 items derived from a thematic analysis of in-depth interviews of 31 parents and 2 FGDs with eight parents admitted at CPC and eight mental health professionals experienced in child and adolescent psychiatry. It is a 5-point Likert scale. The tool showed a high item-level Content Validity Index (I-CVI range: 0.8-1.0) and a high Scale Content Validity Index (S-CVI = 0.91). The test-retest method was used to calculate reliability. A total of 38 parents were included in the test-retest method. The mother and father's mean age and standard deviation (SD) scores were 39.7 ± 5.8 and 45.1 ± 4.9, respectively. Most parents' education level is above graduation (mothers: 86.8% and fathers: 86%). Around 63% of mothers were housewives and had normal pregnancies. The single measure two-way mixed absolute agreement intraclass correlation coefficient (ICC) value for 38 subjects was 0.96 with a 95% confidence interval (CI: 0.90-0.95), indicating high reliability. PCCMI has high I-CVI and S-CVI and good ICC test-retest reliability. This tool will help mental health professionals in the formal assessment of parental care for children with mental illness and provide the intervention to address issues related to the same.

Comparison of semi-automatic and manual segmentation methods for tumor delineation on head and neck squamous cell carcinoma (HNSCC) positron emission tomography (PET) images

Objective. Accurate and reproducible tumor delineation on positron emission tomography (PET) images is required to validate predictive and prognostic models based on PET radiomic features. Manual segmentation of tumors is time-consuming whereas semi-automatic methods are easily implementable and inexpensive. This study assessed the reliability of semi-automatic segmentation methods over manual segmentation for tumor delineation in head and neck squamous cell carcinoma (HNSCC) PET images. Approach. We employed manual and six semi-automatic segmentation methods (just enough interaction (JEI), watershed, grow from seeds (GfS), flood filling (FF), 30% SUVmax and 40%SUVmax threshold) using 3D slicer software to extract 128 radiomic features from FDG-PET images of 100 HNSCC patients independently by three operators. We assessed the distributional properties of all features and considered 92 log-transformed features for subsequent analysis. For each paired comparison of a feature, we fitted a separate linear mixed effect model using the method (two levels; manual versus one semi-automatic method) as a fixed effect and the subject and the operator as the random effects. We estimated different statistics—the intraclass correlation coefficient agreement (aICC), limits of agreement (LoA), total deviation index (TDI), coverage probability (CP) and coefficient of individual agreement (CIA)—to evaluate the agreement between the manual and semi-automatic methods. Main results. Accounting for all statistics across 92 features, the JEI method consistently demonstrated acceptable agreement with the manual method, with median values of aICC = 0.86, TDI = 0.94, CP = 0.66, and CIA = 0.91. Significance. This study demonstrated that JEI method is a reliable semi-automatic method for tumor delineation on HNSCC PET images.

Reliability and measurement error of a maximal voluntary toe plantarflexion measurement process

Despite the growing interest, information regarding the psychometric properties of maximal voluntary isometric toe plantarflexion force and rate of force development (RFD) is lacking. Hence, we investigate the test-retest reliability and measurement error of these outcome measurement instruments measured with a custom-built dynamometer. Twenty-six healthy adults participated in a crossed design with four sessions separated by 5–7 days. RFD was quantified using manual onset and calculating the impulse and the slope in the following time windows: 0–50 ms, 0–100 ms, 0–150 ms, 0–200 ms, 0–250 ms. We estimated the systematic bias of the mean, the intraclass correlation coefficient (ICC) and standard error of measurement (SEM) from the agreement and consistency models. The ICC and the SEM agreement for maximal voluntary isometric toe plantarflexion force along the perpendicular axis were respectively 0.87 (95%CI: 0.76, 0.93) and 27 N (22, 32), while along the resultant of the perpendicular and anterior posterior axis they were 0.85 (0.73, 0.92) and 29 N (23, 35). The results of the consistency model were similar as the estimated variance for session was closer to zero. A systematic bias of the mean between session 1 and 3 was found. For the RFD variables, the ICC agreement ranged from 0.35 to 0.65. The measurement process was found to be reliable to assess maximal voluntary isometric toe plantarflexion force but not RFD. However, a familiarization session is mandatory and these results need to be confirmed in less coordinated (e.g. aging population) individuals.

Further development and validation of the Multimorbidity Treatment Burden Questionnaire (MTBQ)

ObjectivesTo undertake further psychometric testing of the Multimorbidity Treatment Burden Questionnaire (MTBQ) and examine whether reversing the scale reduced floor effects.DesignSurvey.SettingUK primary care.ParticipantsAdults (≥18 years) with three or more long-term...

Quantitative MRI assessment of joint effusion using T2-relaxometry at 3 Tesla: a feasibility and reproducibility study

ObjectiveT2-relaxometry could differentiate between physiological and haemorrhagic joint effusion (≥ 5% blood) in vitro. Are quantitative T2-relaxation time measurements of synovial fluid feasible and reproducible in vivo in clinically bleed-free joints of men with haemophilia?Materials and methodsIn this cross-sectional study, we measured T2-relaxation times of synovial fluid in clinically bleed-free ankles, knees or elbows of men with severe haemophilia A using a T2-mapping sequence (duration ≤ 7 min) at 3 Tesla MRI. Manual and circular regions of interest (ROI) were drawn in the synovial fluid of each joint by two independent observers to measure T2-relaxation times. Measurement feasibility was expressed as the success rate of the measurements by both observers. The interobserver and intraobserver reproducibility of the measurements were evaluated by the intraclass correlation coefficient of absolute agreement (ICC) and the limits of agreement (LoA) from Bland Altman analysis.ResultsWe evaluated 39 clinically bleed-free joints (11 ankles, 12 knees, 16 elbows) of 39 men (median age, 24 years; range 17–33) with severe haemophilia A. The success rate of the T2-measurements was ≥ 90%. Interobserver reliability was good to excellent (manual ROI: ICC = 0.92, 95% CI 0.76–0.97; circular ROI: ICC = 0.82, 95% CI 0.66–0.91) and interobserver agreement was adequate (manual ROI: LoA = 71 ms; circular ROI: LoA = 146 ms). Intraobserver reliability was good to excellent (manual ROI: ICC = 0.78, 95% CI − 0.06–0.94; circular RO: ICC = 0.99, 95% CI 0.98–0.99) and intraobserver agreement was good (manual ROI: LoA = 63 ms; circular ROI: LoA = 41 ms).ConclusionT2-relaxometry of synovial fluid in haemophilia patients is feasible with good interobserver and intraobserver reproducibility.

Comparison of injury severity scores derived from ICD-10-AM codes with trauma registry derived scores: A study from New Zealand

IntroductionVarious attempts at automation have been made to reduce the administrative burden of manually assigning Abbreviated Injury Severity (AIS) codes to derive Injury Severity Scores (ISS) in trauma registry data. The accuracy of the resulting measures remains unclear, especially in the New Zealand (NZ) context. The aim of this study was to compare ISS derived from hospital discharge International Classification of Diseases Australian Modification (ICD-10-AM) codes with ISS recorded in the NZ Trauma Registry (NZTR). MethodsIndividuals admitted to hospital and enrolled in the NZTR between 1 December 2016 and 30 November 2018 were included. ISS were calculated using a modified ICD to AIS mapping tool. The agreement between both methods for raw scores was assessed by the Intraclass Correlation Coefficient (ICC), and for categorical scores the Kappa and weighted Kappa index were used. Analysis was conducted by gender, age, ethnicity, and mechanism of injury. Results3,156 patients fulfilled the inclusion criteria. The ICC for agreement between the methods was poor (0.40, 95 % CI: 0.37–0.43). The Kappa index indicated slight agreement between both methods when using a cut-off value of 12 (0.06; 95 % CI: 0.01–0.12) and 15 (0.13 6; 95 % CI: 0.09–0.17). ConclusionAlthough the overall agreement between NZTR-ISS and ICD-ISS was slight, ICD-derived scores may be useful to describe injury patterns and for body region-specific estimations when manually coded ISS are not available.

Novel AI-Based Algorithm for the Automated Measurement of Cervical Sagittal Balance Parameters. A Validation Study on Pre- and Postoperative Radiographs of 129 Patients.

Retrospective, mono-centric cohort research study. The analysis of cervical sagittal balance parameters is essential for preoperative planning and dependent on the physician's experience. A fully automated artificial intelligence-based algorithm could contribute to an objective analysis and save time. Therefore, this algorithm should be validated in this study. Two surgeons measured C2-C7 lordosis, C1-C7 Sagittal Vertical Axis (SVA), C2-C7-SVA, C7-slope and T1-slope in pre- and postoperative lateral cervical X-rays of 129 patients undergoing anterior cervical surgery. All parameters were measured twice by surgeons and compared to the measurements by the AI algorithm consisting of 4 deep convolutional neural networks. Agreement between raters was quantified, among other metrics, by mean errors and single measure intraclass correlation coefficients for absolute agreement. ICC-values for intra- (range: .92-1.0) and inter-rater (.91-1.0) reliability reflect excellent agreement between human raters. The AI-algorithm could determine all parameters with excellent ICC-values (preop:0.80-1.0; postop:0.86-.99). For a comparison between the AI algorithm and 1 surgeon, mean errors were smallest for C1-C7 SVA (preop: -.3mm (95% CI:-.6 to -.1mm), post: .3mm (.0-.7mm)) and largest for C2-C7 lordosis (preop:-2.2° (-2.9 to -1.6°), postop: 2.3°(-3.0 to -1.7°)). The automatic measurement was possible in 99% and 98% of pre- and postoperative images for all parameters except T1 slope, which had a detection rate of 48% and 51% in pre- and postoperative images. This study validates that an AI-algorithm can reliably measure cervical sagittal balance parameters automatically in patients suffering from degenerative spinal diseases. It may simplify manual measurements and autonomously analyze large-scale datasets. Further studies are required to validate the algorithm on a larger and more diverse patient cohort.

Interobserver variability of assessing body condition scores and muscle condition scores in a population of 43 active working explosive detection dogs.

This study aimed to evaluate the agreement between explosive detection dog (EDD) handlers and a team of veterinarians in assessing body condition score (BCS) and muscle condition score (MCS), hypothesizing significant BCS differences between handlers and veterinarians, and no significant MCS differences in healthy active duty EDDs. This prospective study analyzed variance and inter-rater intraclass correlation coefficients (ICC) of agreement within BCS and MCS assessments collected from the 43 EDDs by four blinded graders; the EDDs' respective handler and three veterinarians with varying levels of veterinary expertise. The results of the study showed that 74.4% of the EDD population was graded as ideal BCS (4 or 5 out of 9) by the handlers compared to 67.44% by the members of the veterinary team; however, the graders scored different subsets of individual EDDs as ideal. Normal MCS (3 out of 3) was assessed in 86.05% (n = 37) of EDDs by the handlers versus in 70.54% by the veterinary team. This study highlights the importance of standardized training and guidelines for BCS and MCS assessments in working dogs to improve agreement between all members of the healthcare team.

Validation of algorithms for calculating spatiotemporal gait parameters during continuous turning using lumbar and foot mounted inertial measurement units

Spatiotemporal gait parameters such as step time and walking speed can be used to quantify gait performance and determine physical function. Inertial measurement units (IMUs) allow for the measurement of spatiotemporal gait parameters in unconstrained environments but must be validated against a gold standard.While many IMU systems and algorithms have been validated during treadmill walking and overground walking in a straight line, fewer studies have validated algorithms during more complex walking conditions such as continuous turning in different directions.This study explored the concurrent validity in a population of healthy adults (range 26–52 years) of three different algorithms using lumbar and foot mounted IMUs to calculate spatiotemporal gait parameters: two methods utilizing an inverted pendulum model, and one method based on strapdown integration. IMU data was compared to a Vicon twelve-camera optoelectronic system, using data collected from 9 participants performing straight walking and continuous walking trials at different speeds, resulting in 162 walking trials in total. Intraclass correlation coefficients (ICCA,1) for absolute agreement were calculated between the algorithm outputs and Vicon output.Temporal parameters were comparable in all methods and ranged from moderate to excellent, except double support time which was poor. Strapdown integration performed better for estimating spatial parameters than pendulum models during straight walking, but worse during turning. Selecting the most appropriate model should take into consideration both speed and walking condition.

Concurrent Validity and Reliability of In-Field Vertical Jump Performance Measures on Sand Surfaces.

Wirtz, S, Julian, R, Schmale, R, and Eils, E. Concurrent validity and reliability of in-field vertical jump performance measures on sand surfaces. J Strength Cond Res 38(4): 687-693, 2024-Beach sports, such as handball, volleyball, and soccer, involve dynamic actions, primarily jumps. Nonetheless, there is a need for more established measurement devices that are both valid and feasible in accurately evaluating jumping performance on sand surfaces. This study aims to assess the reliability and concurrent validity of MyJump2 and an inertial measurement unit (VERT) for quantifying countermovement jump performance on the sand. Twenty-one subjects performed 5 countermovement jumps on the sand. Jumping height was measured simultaneously by a force plate, VERT, and MyJump2. The VERT and MyJump2 reliability measures were evaluated using the intraclass correlation coefficients (ICC) for both absolute agreement (ICC 2,1 ) and consistency (ICC 3.1 ). The day-to-day reliability of both devices and the interrater and intrarater reliability of the MyJump2 were assessed. The concurrent validity of the VERT and MyJump2 was assessed using ICCs 2.1, 3.1 and Bland-Altman plots. A near-perfect agreement was seen for intrarater (ICC = 0.98) and interrater (ICC = 0.98) reliability of the MyJump2. Moreover, a near-perfect agreement was also observed when comparing the MyJump2 with the force plate (ICC = 0.92), with a mean bias of -2.31 cm. The VERT device only demonstrated a moderate agreement (ICC = 0.55) with a mean bias of 6.31 cm compared with the force plate, despite showing good day-to-day reliability (ICC = 0.79). The study's results indicate that MyJump2 is valid and reliable for assessing jump heights on sand surfaces, whereas VERT cannot be recommended.

Assessing the robustness of muscle strength and physical performance measures in women older than 40 years: a test-retest reliability study.

This study investigated the test-retest reliability and minimal detectable change (MDC) of muscle strength and physical performance tests in women older than 40 years. A total of 113 women, aged 58 ± 11 years (82% postmenopausal women), with a body mass index of 28.5 ± 5.8 kg/m 2 , participated in this study. One-repetition maximum (1RM) tests were conducted for seven strength exercises: 45° leg press, bench press, leg extension, pec deck, wide-grip lateral pull-down, leg curl, and seated cable row, with a retest after 48 hours. Timed Up and Go (TUG), 6-minute walk, and 30-second sit-to-stand tests were performed on the same day and retested after 72 hours. The tests and retests were administered by the same evaluators. Relative reliability (consistency of participant rank between test-retest) was assessed using the intraclass correlation coefficient for consistency and agreement, and absolute reliability (precision of score) was assessed using the MDC based on the standard error of prediction. The 1RM and performance tests exhibited excellent reliability: 45° leg press (consistency, 0.99; agreement, 0.98), bench press (consistency, 0.96; agreement, 0.96), leg extension (consistency, 0.93; agreement, 0.91), pec deck (consistency, 0.90; agreement, 0.88), wide-grip lateral pull-down (consistency, 0.91; agreement, 0.89), leg curl (consistency, 0.84; agreement, 0.83), seated cable row (consistency, 0.92; agreement, 0.94), TUG (consistency, 0.87; agreement, 0.87), 6-minute walk (consistency, 0.96; agreement, 0.95), and 30-second sit to stand (consistency, 0.85; agreement, 0.80). These tests showed meaningful MDC values, particularly for the 1RM test performed on the machine and the 30-second sit-to-stand test: 45° leg press, 22.2 kg (15%); bench press, 4.9 kg (19%); leg extension, 9.2 kg (35%); pec deck, 8.8 kg (44%); wide-grip lateral pull-down, 9.4 kg (31%); leg curl, 9.2 kg (51%); seated cable row, 11.4 kg (29%); TUG, 1.4 seconds (20%); 6-minute walk, 50.9 m (10%); and 30-second sit to stand, 4.6 reps (30%). Although the muscle strength and physical performance tests demonstrate excellent relative reliability in women older than 40 years, they exhibit low absolute reliability, particularly the 1RM test performed on the machine and the 30-second sit-to-stand test. Therefore, although these tests show good consistency in the positioning of women within the group after repeated measures, their precision measure is relatively low (high fluctuation). Changes in these tests that are smaller than the MDC may not indicate real changes in women at middle age and older.

Episiotomy: Are Indian Obstetricians Getting the Angle Right?

BackgroundA mediolateral episiotomy is recommended when indicated at a 60° angle at crowning, to avoid obstetric anal sphincter injuries (OASIs) by episiotomies angled too close or distant to the anus. This study surveyed obstetricians in India regarding the recommended episiotomy angle and their ability to correctly draw the angle.MethodsWorkshops were conducted in India to share knowledge in the prevention and repair of OASIs. A questionnaire was distributed prior to the workshop. Participants were asked to describe the recommended episiotomy angle and to draw this on a paper replica of the perineum. The intra-class correlation coefficient (ICC) was calculated to assess the inter-rater reliability between the angle stated and drawn. A 2° difference was deemed acceptable. Standard errors of measurement (SEM) were calculated to measure the range of error of each measurement.ResultsOne hundred and forty doctors participated. 47.9% described the angle of an episiotomy to be 60°. Only 2.2% drew an angle of 60°, but 8.7% (n = 12) drew between 58 and 62°. Only 5.8% (n = 6) of doctors correctly drew the episiotomy angle they described. There was poor agreement ICC = 0.18 (− 0.01 to 0.36) with a SEM of ± 12.2°ConclusionsKnowledge surrounding the recommended episiotomy angle is lacking. Doctors are failing to estimate their desired episiotomy angle. This highlights the need for national guidelines, the creation and validation of structured training programmes to improve accuracy, or using fixed-angle devices such as the EPISCISSORS-60 or other proven measurement aids to minimise preventable harm due to human error.

Collecting evidence of validity for an assessment tool for Norwegian medical students’ non-technical skills (NorMS-NTS): usability and reliability when used by novice raters

BackgroundThe NorMS-NTS tool is an assessment tool for assessing Norwegian medical students’ non-technical skills (NTS). The NorMS-NTS was designed to provide student feedback, training evaluations, and skill-level comparisons among students at different study sites. Rather than requiring extensive rater training, the tool should capably suit the needs of busy doctors as near-peer educators. The aim of this study was to examine the usability and preliminary assess validity of the NorMS-NTS tool when used by novice raters.MethodsThis study focused on the usability of the assessment tool and its internal structure. Three raters used the NorMS-NTS tool to individually rate the team leader, a medical student, in 20 video-recorded multi-professional simulation-based team trainings. Based on these ratings, we examined the tools’ internal structure by calculating the intraclass correlation coefficient (ICC) (version 3.1) interrater reliability, internal consistency, and observability. After the rating process was completed, the raters answered a questionnaire about the tool’s usability.ResultsThe ICC agreement and the sum of the overall global scores for all raters were fair: ICC (3,1) = 0.53. The correlation coefficients for the pooled raters were in the range of 0.77–0.91. Cronbach’s alpha for elements, categories and global score were mostly above 0.90. The observability was high (95%-100%). All the raters found the tool easy to use, none of the elements were redundant, and the written instructions were helpful. The raters also found the tool easier to use once they had acclimated to it. All the raters stated that they could use the tool for both training and teaching.ConclusionsThe observed ICC agreement was 0.08 below the suggested ICC level for formative assessment (above 0.60). However, we know that the suggestion is based on the average ICC, which is always higher than a single-measure ICC. There are currently no suggested levels for single-measure ICC, but other validated NTS tools have single-measure ICC in the same range. We consider NorMS-NTS as a usable tool for formative assessment of Norwegian medical students’ non-technical skills during multi-professional team training by raters who are new to the tool. It is necessary to further examine validity and the consequences of the tool to fully validate it for formative assessments.

Deep-learning based analysis of intracoronary optical coherence tomography images: detection and characterization of lipid plaques

Abstract Background Intracoronary optical coherence tomography (OCT) enables in vivo detection and characterization of atherosclerotic disease. However, manual assessment of OCT images is time-consuming and subject to intra- and interobserver variability. Therefore, automated and trustworthy methods for plaque assessment are warranted. Purpose To develop and validate an artificial intelligence based algorithm for detection and characterization of lipid plaques on OCT images. Methods From the prospective, observational PECTUS-obs study, we studied a representative subset of OCT pullbacks performed in fractional flow reserve-negative non-culprit lesions in patients presenting with myocardial infarction. Manual segmentation of cross-sectional frames (e.g. lumen, tunica intima including the fibrous cap, tunica media and lipid pools) was performed by trained experts. Pullbacks were randomly divided into a training and test set. For automated segmentation and analysis, a two-dimensional no-new U-shaped neural network (nnUNet) was constructed with 5-fold cross validation. To test the diagnostic performance of the nnUnet for detection of lipid plaques, sensitivity, specificity and Cohen’s kappa were calculated. As for plaque characterization, the lipid arc and minimal fibrous cap thickness were measured manually in the test set on each frame containing a lipid plaque. Values of lipid arc and minimal fibrous cap thickness obtained following automated assessment were compared to manual analysis using the intraclass correlation coefficient (ICC) for absolute agreement. Results In total, 1215 frames were used for training and 162 frames for testing of the nnUNet. Lipid plaques were present in 77 frames (47.5%) in the test set. Substantial agreement (κ=0.79) for detection of a lipid plaque was achieved with a sensitivity and specificity of 96.1% and 83.5%, respectively. As for plaque characterization, we found good reliability for lipid arc assessment (ICC 0.88, mean difference 6±32º) and moderate reliability for assessment of the minimal fibrous cap thickness (ICC 0.54, mean difference 23±140µm). Conclusion We developed a deep learning algorithm for automated analysis of intracoronary OCT images, that was able to accurately detect lipid plaques. Moreover, the algorithm showed promising results in terms of plaque characterization. With further refinements, the developed algorithm has the potential to reduce the time required for OCT interpretation, ultimately enabling more efficient, real-time use of this imaging modality in clinical practice.

12 The Development of a Pediatric Metamemory Questionnaire and Scoring Procedure

Objective:To create a standardized scoring procedure to evaluate open-ended responses as part of a novel questionnaire (Measure of Metamemory; MoM) designed to assess declarative metamemory in youth. Metamemory is an aspect of metacognition that is one’s knowledge of the factors related to storage and retrieval of information (Flavell 1971; Kreutzer et al., 1975), and includes both declarative metamemory (i.e., one’s knowledge about factors influencing memory) and procedural metamemory (i.e., one’s understanding of their own memory performance).Participants and Methods:Fourteen short vignettes related to memory were administered to 100 participants (age 6-12) with questions such as, “Two children hear a story they must remember. The first person is 5 years old. The second person is 12 years old. Who is most likely to remember it best?” After answering, they were then prompted to explain their answer (i.e., “Why?”) and their responses were recorded verbatim. To develop standardized and objective criteria for each of the 14 open-ended responses, responses from a subsample of 20 youth were collectively examined by the study team and a scoring structure similar to open-ended items on common intelligence tests (e.g., WISC-V/WAIS-IV) was created. Two points (full credit) were awarded for complete and thorough understanding of memory processes related to the question; 1 point was given for partially accurate or incomplete understanding of the related memory process; and 0 points for an inability to correctly express an understanding of relevant memory concepts. This scoring guide was then applied independently by each of the six raters to an additional 25 participants (ages 6-12 mean age (SD)). To assess the interrater reliability of this 3-point ordinal scoring system, we examined both Fleiss’ kappa and 2-way random-effects, single-rater, absolute agreement Intra-Class Correlations (ICC).Results:Across the six independent raters, reliability coefficients for each of the 14 items ranged from (Fleiss') k =. 277 to .792 (ICC ranged from .481 to .880). Of these 14 items, the kappa value was classified (using interpretation rules for Cohen’s kappa) as “substantial” for 10 items, “moderate” for 3 items, and “fair” for 1 item. Based on these lower inter-rater reliabilities, two items were subsequently removed from the measure to create the 12-item open-ended measure of metamemory, the MoM-12, with reliable scoring for youth as young as 6 years old.Conclusions:A consensus process established a quantifiable scoring procedure to assess open-ended responses related to youth’s knowledge of memory (e.g., metamemory). Reliability metrics identified acceptable interrater reliabilities in 12 of the 14 original items. Further examination of psychometric properties, including internal consistency and lines of evidence for validity is needed. The successful crafting of a scoring procedure is a first step towards developing a psychometrically sound measure (the MoM) to evaluate and study metamemory concepts objectively and reliably in youth.

Development and Validation of a Formative Assessment Tool for Nephrology Fellows' Clinical Reasoning.

Diagnostic errors are commonly driven by failures in clinical reasoning. Deficits in clinical reasoning are common among graduate medical learners, including nephrology fellows. We created and validated an instrument to assess clinical reasoning in a national cohort of nephrology fellows and established performance thresholds for remedial coaching. Experts in nephrology education and clinical reasoning remediation designed an instrument to measure clinical reasoning through a written patient encounter note from a web-based, simulated AKI consult. The instrument measured clinical reasoning in three domains: problem representation, differential diagnosis with justification, and diagnostic plan with justification. Inter-rater reliability was established in a pilot cohort ( n =7 raters) of first-year nephrology fellows using a two-way random effects agreement intraclass correlation coefficient model. The instrument was then administered to a larger cohort of first-year fellows to establish performance standards for coaching using the Hofstee method ( n =6 raters). In the pilot cohort, there were 15 fellows from four training program, and in the study cohort, there were 61 fellows from 20 training programs. The intraclass correlation coefficients for problem representation, differential diagnosis, and diagnostic plan were 0.90, 0.70, and 0.50, respectively. Passing thresholds (% total points) in problem representation, differential diagnosis, and diagnostic plan were 59%, 57%, and 62%, respectively. Fifty-nine percent ( n =36) met the threshold for remedial coaching in at least one domain. We provide validity evidence for a simulated AKI consult for formative assessment of clinical reasoning in nephrology fellows. Most fellows met criteria for coaching in at least one of three reasoning domains, demonstrating a need for learner assessment and instruction in clinical reasoning.

IODA-shoulder®, a 3-dimensional automated software for glenoid bone loss quantification in shoulder instability. A validation study

BackgroundGlenoid bone loss (GBL) calculation in patients with shoulder instability has a wide variability in methods and their reliability. The purpose of this study was to describe and validate a new semi-automated software developed to improve GBL calculation using a 3D imaging modeling (IODA-shoulder) and to compare the method to the PICO area method. Patients and methodsA semi-automated software to assess GBL was preliminarily developed and validated on 7 fresh frozen specimens (scapulae with artificially created glenoid defect), using water displacement method. Afterwards, the software was retrospectively used on CT images of 20 patients affected by recurrent shoulder dislocation. Inclusion criteria were: unilateral dislocation, minimum 2 dislocation episodes. Exclusion criteria were bone reconstruction of the glenoid, failed shoulder stabilization, bilateral dislocation, shoulder arthritis. Three-dimensional computed tomography images of bilateral shoulder were retrieved for each patient. Two methods to determine GBL were compared. The PICO surface area method and the new IODA method. We assessed the intra- and inter-rater reliability of the two methods with the Intraclass Correlation Coefficient (ICC), the Bland-Alman analysis, and Lin's concordance correlation coefficient (CCC). ResultsWe did not find a statistically significant difference between the mean volumes calculated with PICO and IODA methods, respectively 914 vs. 815 mm3, p=0.155. The analysis carried out by using the traditional PICO method showed a lower concordance rate among four observers than the higher concordance found using IODA method, regardless of the size and the location of the defect. The ICC agreement with PICO was significantly lower than with IODA (0.76 vs. 0.97). We found a poor CCC with PICO (from 0.65 to 0.81) and a substantial one with IODA (from 0.96 to 0.98). DiscussionThe intra- and inter-rater reliability using IODA method is significantly better than PICO method. The assessment of GBL using IODA method is time saving, avoiding significant inter- and intra-observer variation, mainly due to individual skill and experience in the method. Level of evidenceIV, experimental study.