Median Intraclass Correlation Research Articles

Development and benchmarking of a Deep Learning-based MRI-guided gross tumor segmentation algorithm for Radiomics analyses in extremity soft tissue sarcomas

BackgroundVolume of interest (VOI) segmentation is a crucial step for Radiomics analyses and radiotherapy (RT) treatment planning. Because it can be time-consuming and subject to inter-observer variability, we developed and tested a Deep Learning-based automatic segmentation (DLBAS) algorithm to reproducibly predict the primary gross tumor as VOI for Radiomics analyses in extremity soft tissue sarcomas (STS). MethodsA DLBAS algorithm was trained on a cohort of 157 patients and externally tested on an independent cohort of 87 patients using contrast-enhanced MRI. Manual tumor delineations by a radiation oncologist served as ground truths (GTs). A benchmark study with 20 cases from the test cohort compared the DLBAS predictions against manual VOI segmentations of two residents (ERs) and clinical delineations of two radiation oncologists (ROs). The ROs rated DLBAS predictions regarding their direct applicability. ResultsThe DLBAS achieved a median dice similarity coefficient (DSC) of 0.88 against the GTs in the entire test cohort (interquartile range (IQR): 0.11) and a median DSC of 0.89 (IQR 0.07) and 0.82 (IQR 0.10) in comparison to ERs and ROs, respectively. Radiomics feature stability was high with a median intraclass correlation coefficient of 0.97, 0.95 and 0.94 for GTs, ERs, and ROs, respectively. DLBAS predictions were deemed clinically suitable by the two ROs in 35% and 20% of cases, respectively. ConclusionThe results demonstrate that the DLBAS algorithm provides reproducible VOI predictions for radiomics feature extraction. Variability remains regarding direct clinical applicability of predictions for RT treatment planning.

Inter- and Intra-Patient Repeatability of Radiomic Features from Multiparametric Whole-Body MRI in Patients with Metastatic Prostate Cancer.

(1) Background: We assessed the test-re-test repeatability of radiomics in metastatic castration-resistant prostate cancer (mCPRC) bone disease on whole-body diffusion-weighted (DWI) and T1-weighted Dixon MRI. (2) Methods: In 10 mCRPC patients, 1.5 T MRI, including DWI and T1-weighted gradient-echo Dixon sequences, was performed twice on the same day. Apparent diffusion coefficient (ADC) and relative fat-fraction-percentage (rFF%) maps were calculated. Per study, up to 10 target bone metastases were manually delineated on DWI and Dixon images. All 106 radiomic features included in the Pyradiomics toolbox were derived for each target volume from the ADC and rFF% maps. To account for inter- and intra-patient measurement repeatability, the log-transformed individual target measurements were fitted to a hierarchical model, represented as a Bayesian network. Repeatability measurements, including the intraclass correlation coefficient (ICC), were derived. Feature ICCs were compared with mean ADC and rFF ICCs. (3) Results: A total of 65 DWI and 47 rFF% targets were analysed. There was no significant bias for any features. Pairwise correlation revealed fifteen ADC and fourteen rFF% feature sub-groups, without specific patterns between feature classes. The median intra-patient ICC was generally higher than the inter-patient ICC. Features that describe extremes in voxel values (minimum, maximum, range, skewness, and kurtosis) showed generally lower ICCs. Several mostly shape-based texture features were identified, which showed high inter- and intra-patient ICCs when compared with the mean ADC or mean rFF%, respectively. (4) Conclusions: Pyradiomics texture features of mCRPC bone metastases varied greatly in inter- and intra-patient repeatability. Several features demonstrated good repeatability, allowing for further exploration as diagnostic parameters in mCRPC bone disease.

Abstract 2174: Sample size estimations based on human microbiome temporal stability over six months: A shallow shotgun metagenome sequencing analysis

Abstract Introduction: Many biological factors can influence the human microbiome; thus, it is necessary to reasonably estimate sample size to have adequate power for future population-based studies. Methods: We evaluated the temporal stability of fecal microbial diversity, species level composition, and genes and functional pathways based on shallow shotgun metagenome sequencing from a population-based study in Costa Rica with two fecal samples separated by an interval of six months. We quantified the biological variability over six months with intraclass correlation coefficients (ICC). We then estimated the numbers of cases, assuming 1:1 or 1:3 matched case-control study, required to detect an association at significance levels of 0.05 and 0.001 with 80% power, based on the number of fecal specimens collected per participant. Results: For most alpha and beta diversity metrics, the temporal stability of fecal microbiome in samples collected at six months of interval was low to moderate, with ICCs of 0.6 or less. We observed heterogeneity in temporal stability for the proportions of species, genes and pathways, with ICCs varying between 0.0 and 0.9. For most microbiome measures from a single fecal collection, assuming an equal number of cases and controls at significance level of 0.05 (for alpha and beta diversity) or 0.001 (for species, genes and pathways), detecting an odds ratio of 1.5 per standard deviation of microbiome metric would require hundreds to thousands of cases. Specifically, detecting an association between alpha/beta diversity metrics and an outcome would require between 1,000-5,000 cases. In addition, for low prevalent species (5% - 10%, median ICC = 0.09), 15,102 cases would be required; in contrast, for species with high prevalence (>75%, median ICC = 0.41) 3,527 cases would be necessary; the same applies to genes and pathways. For an odds ratio of 1.5, assuming a 1:3 matched case-control study based on one fecal specimen per subject, 10,068 cases would be required for low prevalent species and 2,351 cases for species with high prevalence. Using the same study setting with multiple specimens per subject over time, the required sample size would be lower. Indeed, detecting an odds ratio of 1.5 for low prevalent species would require 15,102 cases with one specimen, 8,267 cases with two specimens, and 5,989 cases with three specimens. Conclusion: Our calculations suggest that to detect modest disease associations we would need a substantial number of cases. Repeated prediagnostic samples could decrease the number of subjects required to detect these associations as well as matching cases to a great number of controls. Citation Format: Semi Zouiouich, Smriti Karwa, Yunhu Wan, Emily Vogtmann, Carolina Porras, Christian C. Abnet, Jianxin Shi, Rashmi Sinha. Sample size estimations based on human microbiome temporal stability over six months: A shallow shotgun metagenome sequencing analysis [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 2174.

Robotic-assisted Total Knee Arthroplasty Technology Provides a Repeatable and Reproducible Method of Assessing Soft Tissue Balance.

Soft-tissue balancing is an important factor in primary total knee arthroplasty (TKA), with 30 to 50% of TKA revisions attributed to technical operative factors including soft-tissue balancing. Robotic-assisted TKA (RATKA) offers opportunities for improved soft-tissue balancing methods. This study aimed to evaluate the repeatability and reproducibility of ligamentous laxity assessments during RATKA using a digital tensioner.Three experienced RATKA surgeons assessed preresection and trialing phases of 12 human cadaveric knees with varying degrees of arthritis. Ligamentous laxity was assessed with manual varus and valgus stresses in extension and flexion, with a digital tensioner providing feedback on the change of laxity displacement. Intraclass correlation coefficient (ICC) analyses were used to determine the repeatability within a single surgeon and reproducibility between the three surgeons.The results showed excellent repeatability and reproducibility in ligamentous laxity assessment during RATKA. Surgeons had excellent repeatability for preresection and trialing assessments, with median ICC values representing excellent reproducibility between surgeons. Surgeons were repeatable within 1 or 1.5 mm for preresection and trialing assessments. On average, the variation within a surgeon was 0.33 ± 0.26 mm during preresection and 0.29 ± 0.28 mm during trialing. When comparing surgeons to each other, they were reproducible within an average of 0.69 ± 0.33 mm for preresection and 0.65 ± 0.31 mm for trialing.This study demonstrated the reliability of robotic-assisted soft-tissue balancing techniques, providing control over ligamentous laxity assessments, and potentially leading to better patient outcomes. The digital tensioner used in this study provided excellent repeatability and reproducibility in ligamentous laxity assessment during RATKA, highlighting the potential benefits of incorporating robotics in TKA procedures.

Multi-sequence MRI radiomics of colorectal liver metastases: Which features are reproducible across readers?

PurposeTo assess the inter-reader reproducibility of radiomics features on multiple MRI sequences after segmentations of colorectal liver metastases (CRLM). Method30 CRLM (in 23 patients) were manually delineated by three readers on MRI before the start of chemotherapy on the contrast enhanced T1-weighted images (CE-T1W) in the portal venous phase, T2-weighted images (T2W) and b800 diffusion weighted images (DWI). DWI delineations were copied to the ADC-maps. 107 radiomics features were extracted per sequence. The intraclass correlation coefficient (ICC) was calculated per feature. Features were considered reproducible if ICC > 0.9. Results90% of CE-T1W features were reproducible with a median ICC of 0.98 (range 0.76–1.00). 81% of DWI features were robust with median ICC = 0.97 (range 0.38–1.00). The T2W features had a median ICC of 0.96 (range 0.55–0.99) and were reproducible in 80%. ADC showed the lowest number of reproducible features with 58% and median ICC = 0.91 (range 0.38–0.99) When considering the lower bound of the ICC 95% confidence intervals, 58%, 66%, 54% and 29% reached 0.9 for the CE-T1W, DWI, T2W and ADC features, respectively. The feature class with the best reproducibility differed per sequence. ConclusionsThe majority of MRI radiomics features from CE-T1W, T2W, DWI and ADC in colorectal liver metastases were robust for segmentation variability between readers. The CE-T1W yielded slightly better reproducibility results compared to DWI and T2W. The ADC features seem more susceptible to reader differences compared to the other three sequences.

Robustness of [18F]FDG PET/CT radiomic analysis in the setting of drug-induced cardiotoxicity

Background and objectivesStandardization of radiomic data acquisition protocols is still at a very early stage, revealing a strong need to work towards the definition of uniform image processing methodologies The aim of this study is to identify sources of variability in radiomic data derived from image discretization and resampling methodologies prior to image feature extraction. Furthermore, to identify robust potential image-based biomarkers for the early detection of cardiotoxicity. MethodsImage post-acquisition processing, interpolation, and volume of interest (VOI) segmentation were performed. Four experiments were conducted to assess the reliability in terms of the intraclass correlation coefficient (ICC) of the radiomic features and the effects of the variation of voxel size and gray level discretization. Statistical analysis was performed separating the patients according to cardiotoxicity diagnosis. Differences of texture features were studied with Mann-Whitney U test. P-values <0.05 after multiple testing correction were considered statistically significant. Additionally, a non-supervised k-Means clustering algorithm was evaluated. ResultsThe effect of the variation in the voxel size demonstrated a non-dependency relationship with the values of the radiomic features, regardless of the chosen discretization method. The median ICC values were 0.306 and 0.872 for absolute agreement and consistency, respectively, when varying the discretization bin number. The median ICC values were 0.678 and 0.878 for absolute agreement and consistency, respectively, when varying the discretization bin size. A total of 16 first order, 6 Gray Level Co-occurrence Matrix (GLCM), 4 Gray Level Dependence Matrix (GLDM) and 4 Gray Level Run Length Matrix (GLRLM) features demonstrated statistically significant differences between the diagnosis groups for interim scans (P<0.05) for the fixed bin size (FBS) discretization methodology. However, no statistically significant differences between diagnostic groups were found for the fixed bin number (FBN) discretization methodology. Two clusters based on the radiomic features were identified. ConclusionsGray level discretization has a major impact on the repeatability of the radiomic features. The selection of the optimal processing methodology has led to the identification of texture-based patterns for the differentiation of early cardiac damage profiles.

Abstract B072: Reproducibility of plasma metabolome over 1 year in a population-based study of Black breast cancer survivors

Abstract Background: African American/Black women have higher rates of breast cancer (BrCa) mortality and morbidity than White women. The metabolomics approach using blood samples from epidemiological studies has the potential to elucidate pathways or uncover biomarkers for adverse BrCa outcomes. Therefore, understanding the within-person reproducibility of the blood metabolome of BrCa survivors and factors that influence metabolite levels over time is crucial. However, the stability of blood metabolome among Black BrCa survivors is unknown. Objective: This study aimed to estimate the within-person reproducibility of metabolites over 1 year via global untargeted metabolomic profiling of plasma samples collected in the Women’s Circle of Health Follow-Up Study (WCHFS), a population-based longitudinal study of Black BrCa survivors in New Jersey. The study also aims to identify factors associated with reproducibility. Methods: We estimated the metabolite reproducibility among 61 Black women with BrCa diagnosed between 2012 and 2017 in WCHFS. Plasma samples were collected once for each participant at ~2 and 3 years post-diagnosis, respectively, and analyzed at Metabolon via ultrahigh performance liquid chromatography-tandem mass spectrometry. Using a mixed-effects model, we calculated the intraclass correlation coefficients (ICC) for each metabolite by dividing the between-person variance by the total variance (i.e., within- and between-person variances). ICCs were compared by age, obesity status, estrogen receptor (ER) status, and seasons, fasting status and time interval between two visits when blood samples were collected. Statistical significance was determined using the Wilcoxon test. Results: Of the 857 named metabolites identified after excluding those undetectable in ≥80% of samples or with a coefficient of variance≥0.25, the median ICC was 0.58 (interquartile range [IQR]: 0.44-0.70). 16.6% had high within-person reproducibility (ICC≥0.75), spanning across all metabolite classes, while 65.6% had an ICC within 0.4-0.75, and the remaining 17.9% had an ICC&amp;lt;0.4. Significant differences in ICCs were noted by participants’ age (median ICC [IQR; same below]: 0.56 [0.40 - 0.68] for younger women, i.e. ≤ 55.6 years vs. 0.61 [0.44 - 0.73] for older women), obesity status (0.55 [0.40 - 0.68] for non-obese vs. 0.62 [0.46 - 0.75] for obese), and ER status (0.56 [0.41 - 0.68] for ER+ vs. 0.62 [0.45 - 0.79] for ER-). The reproducibility also differed by fasting status (0.63 [0.45 - 0.77] for fasting vs. 0.53 [0.39 - 0.67] for non-fasting), but not the seasons or time interval between blood collections. Conclusion: The within-person reproducibility of plasma metabolites over 1 year among Black breast cancer survivors was generally reasonable, suggesting a single-time-point measurement could be useful in evaluating associations between metabolites and BrCa outcomes. However, the reproducibility varied across metabolites and was influenced by participant characteristics and certain sample collection conditions, which need consideration in future analysis or design. Citation Format: Bo Qin, Madhir Vyas, Steven C. Moore, Xiaoyang Su, Eileen White, Christine B. Ambrosone, Kitaw Demissie, Chi-Chen Hong, Elisa V. Bandera. Reproducibility of plasma metabolome over 1 year in a population-based study of Black breast cancer survivors [abstract]. In: Proceedings of the 16th AACR Conference on the Science of Cancer Health Disparities in Racial/Ethnic Minorities and the Medically Underserved; 2023 Sep 29-Oct 2;Orlando, FL. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2023;32(12 Suppl):Abstract nr B072.

Effects of phase encoding direction on test-retest reliability of human functional connectome

The majority of human connectome studies in the literature based on functional magnetic resonance imaging (fMRI) data use either an anterior-to-posterior (AP) or a posterior-to-anterior (PA) phase encoding direction (PED). However, whether and how PED would affect test-retest reliability of functional connectome is unclear. Here, in a sample of healthy subjects with two sessions of fMRI scans separated by 12 weeks (two runs per session, one with AP, the other with PA), we tested the influence of PED on global, nodal, and edge connectivity in the constructed brain networks. All data underwent the state-of-the-art Human Connectome Project (HCP) pipeline to correct for phase-encoding-related distortions before entering analysis. We found that at the global level, the PA scans showed significantly higher intraclass correlation coefficients (ICCs) for global connectivity compared with AP scans, which was particularly prominent when using the Seitzman-300 atlas (versus the CAB-NP-718 atlas). At the nodal level, regions most strongly affected by PED were consistently mapped to the cingulate cortex, temporal lobe, sensorimotor areas, and visual areas, with significantly higher ICCs during PA scans compared with AP scans, regardless of atlas. Better ICCs were also observed during PA scans at the edge level, in particular when global signal regression (GSR) was not performed. Further, we demonstrated that the observed reliability differences between PEDs may relate to a similar effect on the reliability of temporal signal-to-noise ratio (tSNR) in the same regions (that PA scans were associated with higher reliability of tSNR than AP scans). Averaging the connectivity outcome from the AP and PA scans could increase median ICCs, especially at the nodal and edge levels. Similar results at the global and nodal levels were replicated in an independent, public dataset from the HCP-Early Psychosis (HCP-EP) study with a similar design but a much shorter scan session interval. Our findings suggest that PED has significant effects on the reliability of connectomic estimates in fMRI studies. We urge that these effects need to be carefully considered in future neuroimaging designs, especially in longitudinal studies such as those related to neurodevelopment or clinical intervention.

Exploring regression dilution bias using repeat measurements of 2858 variables in ≤49 000 UK Biobank participants.

Measurement error in exposures and confounders can bias exposure-outcome associations but is rarely considered. We aimed to assess random measurement error of all continuous variables in UK Biobank and explore approaches to mitigate its impact on exposure-outcome associations. Random measurement error was assessed using intraclass correlation coefficients (ICCs) for all continuous variables with repeat measures. Regression calibration was used to correct for random error in exposures and confounders, using the associations of red blood cell distribution width (RDW), C-reactive protein (CRP) and 25-hydroxyvitamin D [25(OH)D] with mortality as illustrative examples. The 2858 continuous variables with repeat measures varied in sample size from 109 to 49 121. They fell into three groups: (i) baseline visit [529 variables; median (interquartile range) ICC = 0.64 (0.57, 0.83)]; (ii) online diet by 24-h recall [22 variables; 0.35 (0.30, 0.40)] and (iii) imaging measures [2307 variables; 0.85 (0.73, 0.94)]. Highest ICCs were for anthropometric and medical history measures, and lowest for dietary and heart magnetic resonance imaging.The ICCs (95% confidence interval) for RDW, CRP and 25(OH)D were 0.52 (0.51, 0.53), 0.29 (0.27, 0.30) and 0.55 (0.54, 0.56), respectively. Higher RDW and levels of CRP were associated with higher risk of all-cause mortality, and higher concentration of 25(OH)D with lower risk. After correction for random measurement error in the main exposure, the associations all strengthened. Confounder correction did not influence estimates. Random measurement error varies widely and is often non-negligible. For UK Biobank we provide relevant statistics and adaptable code to help other researchers explore and correct for this.

Characterisation of Fasting and Postprandial NMR Metabolites: Insights from the ZOE PREDICT 1 Study.

Postprandial metabolomic profiles and their inter-individual variability are not well characterised. Here, we describe postprandial metabolite changes, their correlations with fasting values and their inter- and intra-individual variability, following a standardised meal in the ZOE PREDICT 1 cohort. In the ZOE PREDICT 1 study (n = 1002 (NCT03479866)), 250 metabolites, mainly lipids, were measured by a Nightingale NMR panel in fasting and postprandial (4 and 6 h after a 3.7 MJ mixed nutrient meal, with a second 2.2 MJ mixed nutrient meal at 4 h) serum samples. For each metabolite, inter- and intra-individual variability over time was evaluated using linear mixed modelling and intraclass correlation coefficients (ICC) were calculated. Postprandially, 85% (of 250 metabolites) significantly changed from fasting at 6 h (47% increased, 53% decreased; Kruskal-Wallis), with 37 measures increasing by >25% and 14 increasing by >50%. The largest changes were observed in very large lipoprotein particles and ketone bodies. Seventy-one percent of circulating metabolites were strongly correlated (Spearman's rho >0.80) between fasting and postprandial timepoints, and 5% were weakly correlated (rho <0.50). The median ICC of the 250 metabolites was 0.91 (range 0.08-0.99). The lowest ICCs (ICC <0.40, 4% of measures) were found for glucose, pyruvate, ketone bodies (β-hydroxybutyrate, acetoacetate, acetate) and lactate. In this large-scale postprandial metabolomic study, circulating metabolites were highly variable between individuals following sequential mixed meals. Findings suggest that a meal challenge may yield postprandial responses divergent from fasting measures, specifically for glycolysis, essential amino acid, ketone body and lipoprotein size metabolites.

Within-Subject Variability and the Influence of Exercise Training History on the Resting Plasma Metabolome in Men

This study investigated within-subject variability in the circulating metabolome under controlled conditions, and whether divergent exercise training backgrounds were associated with alterations in the circulating metabolome assessed in resting samples. Thirty-seven men comprising of endurance athletes (END; body mass, 71.0 ± 6.8kg; fat-free mass index, 16.9 ± 1.1kg/m2), strength athletes (STR; 94.5 ± 8.8kg; 23.0 ± 1.8kg/m2), and recreationally active controls (CON; 77.6 ± 7.7kg; 18.1 ± 1.0kg/m2) provided blood samples after an overnight fast on two separate occasions controlled for time of day of sampling, recent dietary intake, time since last meal, and time since last exercise training session. A targeted profile of metabolites, performed using liquid chromatography and mass spectrometry on plasma samples, identified 166 individual metabolites and metabolite features, which were analyzed with intraclass correlation coefficients, a multilevel principal component analysis, and univariate t tests adjusted for multiple comparisons. The median intraclass correlation coefficient was .49, with 46 metabolites displaying good reliability and 31 metabolites displaying excellent reliability. No difference in the abundance of any individual metabolite was identified within groups when compared between visits, but a combined total of 44 metabolites were significantly different (false discovery rate <0.05) between groups (END vs. CON, 42 metabolites; STR vs. CON, 10 metabolites; and END vs. STR, five metabolites). Under similar measurement conditions, the reliability of resting plasma metabolite concentrations varies largely at the level of individual metabolites with ∼48% of metabolites displaying good-to-excellent reliability. However, a history of exercise training was associated with alterations in the abundance of ∼28% of metabolites in the targeted profile employed in this study.

Normative Cerebral Hemodynamics in Middle-aged and Older Adults Using 4D Flow MRI: Initial Analysis of Vascular Aging.

Background Characterizing cerebrovascular hemodynamics in older adults is important for identifying disease and understanding normal neurovascular aging. Four-dimensional (4D) flow MRI allows for a comprehensive assessment of cerebral hemodynamics in a single acquisition. Purpose To establish reference intracranial blood flow and pulsatility index values in a large cross-sectional sample of middle-aged (45-65 years) and older (>65 years) adults and characterize the effect of age and sex on blood flow and pulsatility. Materials and Methods In this retrospective study, patients aged 45-93 years (cognitively unimpaired) underwent cranial 4D flow MRI between March 2010 and March 2020. Blood flow rates and pulsatility indexes from 13 major arteries and four venous sinuses and total cerebral blood flow were collected. Intraobserver and interobserver reproducibility of flow and pulsatility measures was assessed in 30 patients. Descriptive statistics (mean ± SD) of blood flow and pulsatility were tabulated for the entire group and by age and sex. Multiple linear regression and linear mixed-effects models were used to assess the effect of age and sex on total cerebral blood flow and vessel-specific flow and pulsatility, respectively. Results There were 759 patients (mean age, 65 years ± 8 [SD]; 506 female patients) analyzed. For intra- and interobserver reproducibility, median intraclass correlation coefficients were greater than 0.90 for flow and pulsatility measures across all vessels. Regression coefficients β ± standard error from multiple linear regression showed a 4 mL/min decrease in total cerebral blood flow each year (age β = -3.94 mL/min per year ± 0.44; P < .001). Mixed effects showed a 1 mL/min average annual decrease in blood flow (age β = -0.95 mL/min per year ± 0.16; P < .001) and 0.01 arbitrary unit (au) average annual increase in pulsatility over all vessels (age β = 0.011 au per year ± 0.001; P < .001). No evidence of sex differences was observed for flow (β = -1.60 mL/min per male patient ± 1.77; P = .37), but pulsatility was higher in female patients (sex β = -0.018 au per male patient ± 0.008; P = .02). Conclusion Normal reference values for blood flow and pulsatility obtained using four-dimensional flow MRI showed correlations with age. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Steinman in this issue.

Reliability and validity of PRO-CTCAE® daily reporting with a 24-hour recall period

PurposeThe standard recall period for the patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE®) is the past 7 days, but there are contexts where a 24-hour recall may be desirable. The purpose of this analysis was to investigate the reliability and validity of a subset of PRO-CTCAE items captured using a 24-hour recall.Methods27 PRO-CTCAE items representing 14 symptomatic adverse events (AEs) were collected using both a 24-hour recall (24 h) and the standard 7 day recall (7d) in a sample of patients receiving active cancer treatment (n = 113). Using data captured with a PRO-CTCAE-24h on days 6 and 7, and 20 and 21, we computed intra-class correlation coefficients (ICC); an ICC ≥ 0.70 was interpreted as demonstrating high test–retest reliability. Correlations between PRO-CTCAE-24h items on day 7 and conceptually relevant EORTC QLQ-C30 domains were examined. In responsiveness analysis, patients were deemed changed if they had a one-point or greater change in the corresponding PRO-CTCAE-7d item (from week 0 to week 1).ResultsPRO-CTCAE-24h captured on two consecutive days demonstrated that 21 of 27 items (78%) had ICCs ≥ 0.70 (day 6/7 median ICC 0.76), (day 20/21 median ICC 0.84). Median correlation between attributes within a common AE was 0.75, and the median correlation between conceptually relevant EORTC QLQ-C30 domains and PRO-CTCAE-24 h items captured on day 7 was 0.44. In the analysis of responsiveness to change, the median standardized response mean (SRM) for patients with improvement was − 0.52 and that for patients with worsening was 0.71.ConclusionA 24-hour recall period for PRO-CTCAE items has acceptable measurement properties and can inform day-to-day variations in symptomatic AEs when daily PRO-CTCAE administration is implemented in a clinical trial.

Fetal MRI radiomics: non-invasive and reproducible quantification of human lung maturity

ObjectivesTo assess the reproducibility of radiomics features extracted from the developing lung in repeated in-vivo fetal MRI acquisitions.MethodsIn-vivo MRI (1.5 Tesla) scans of 30 fetuses, each including two axial and one coronal T2-weighted sequences of the whole lung with all other acquisition parameters kept constant, were retrospectively identified. Manual segmentation of the lungs was performed using ITK-Snap. One hundred radiomics features were extracted from fetal lung MRI data using Pyradiomics, resulting in 90 datasets. Intra-class correlation coefficients (ICC) of radiomics features were calculated between baseline and repeat axial acquisitions and between baseline axial and coronal acquisitions.ResultsMRI data of 30 fetuses (12 [40%] females, 18 [60%] males) at a median gestational age of 24 + 5 gestational weeks plus days (GW) (interquartile range [IQR] 3 + 3 GW, range 21 + 1 to 32 + 6 GW) were included. Median ICC of radiomics features between baseline and repeat axial MR acquisitions was 0.92 (IQR 0.13, range 0.33 to 1), with 60 features exhibiting excellent (ICC > 0.9), 27 good (> 0.75–0.9), twelve moderate (0.5–0.75), and one poor (ICC < 0.5) reproducibility. Median ICC of radiomics features between baseline axial and coronal MR acquisitions was 0.79 (IQR 0.15, range 0.2 to 1), with 20 features exhibiting excellent, 47 good, 29 moderate, and four poor reproducibility.ConclusionStandardized in-vivo fetal MRI allows reproducible extraction of lung radiomics features. In the future, radiomics analysis may improve diagnostic and prognostic yield of fetal MRI in normal and pathologic lung development.Key Points• Non-invasive fetal MRI acquired using a standardized protocol allows reproducible extraction of radiomics features from the developing lung for objective tissue characterization.• Alteration of imaging plane between fetal MRI acquisitions has a negative impact on lung radiomics feature reproducibility.• Fetal MRI radiomics features reflecting the microstructure and shape of the fetal lung could complement observed-to-expected lung volume in the prediction of postnatal outcome and optimal treatment of fetuses with abnormal lung development in the future.

NIMG-72. VALIDATION OF CONDITIONAL AND SUPER-RESOLUTION GENERATIVE ADVERSARIAL NETWORKS FOR IMPUTATION OF MISSING BRAIN MRI SEQUENCES

Abstract Missing MRI sequences are common in radiology due to differences in protocols, time constraints, and image artifacts. Missing sequences are a nuisance for subsequent algorithms, such as automated segmentation algorithms requiring multiple MRI sequences (e.g., HD-GLIO, which requires T2, T1, post-contrast T1 (T1c), and FLAIR). Accurate tumor segmentation is a fundamental step in extracting radiomics analysis to use, for example, to predict the IDH status of glioma or their overall survival. We validated a conditional (pix2pix) and super-resolution GAN (SOUP-GAN) for imputing MRI sequences on 285 adult glioma with complete MRI sequences available. We simulated missing T1 sequences and subsequently imputed them from T1c using pix2pix and from sagittal T1 using SOUP-GAN. Using HD-GLIO, we obtained the true segmentations from the complete MRI sequences. HD-GLIO was also used after replacing the original T1 sequence with the one imputed with pix2pix and SOUP-GAN. Thus, there were three sets of segmentations for each subject. We compared T2 and T1c radiomics features extracted with Pyradiomics using segmentations obtained with complete data with features extracted from segmentation obtained with the imputed T1 sequences using intraclass correlation coefficient (ICC) and Bland-Altman (BA) analysis. Median (interquartile range) ICC for shape features extracted from T2 and T1c were 0.82(0.76-0.92) using pix2pix and 0.89(0.86-0.94) using SOUP-GAN. Median ICC for first-order-statistics features were 0.90(0.83-0.93) for T2 and 0.90(0.88-0.92) for T1c using pix2pix, and 0.61(0.22-0.82) for T2 and 0.42(0.34-0.57) for T1c using SOUP-GAN. Median ICC for texture features were 0.86(0.82-0.87) for T2 and 0.87(0.79-0.90) for T1c using pix2pix, and 0.69(0.47-0.81) for T2 and 0.76(0.65-0.83) for T1c using SOUP-GAN. We show that T2 and T1c shape features (e.g., volumetric analysis) can be adequately extracted from segmentations obtained with an algorithm requiring multiple MRI sequences using T1 sequences imputed with GANs; however, the effect of imputation differed for first-order-statistics and texture features.

33547 Validation of the Generalized Pustular Psoriasis Physician Global Assessment (GPPGA) scale for assessing severity of GPP

The GPPGA, a novel clinical endpoint for assessing GPP severity, scores pustulation, scaling, and erythema from 0–4. To validate GPPGA as a reliable and consistent measure, reproducibility of scores was evaluated between assessors (interrater) and by the same assessor over time (intrarater). A panel of GPP clinical leaders selected 16 images representing all GPP severities. A cohort (N = 26) of GPP-experienced dermatologists and 3 additional ‘expert raters’ scored GPPGA components during 2 online sessions, separated by ≥10 days. Interrater reliability was assessed by intraclass correlation coefficient (ICC) using an absolute agreement, 2-way mixed-effects model. Intrarater reliability was assessed by ICC using an absolute agreement 2-way random-effects model. ICC thresholds were defined as <0.40, poor; 0.40–0.59, fair; 0.60–0.74, good; 0.75–1.00, excellent. Of 26 dermatologists, 20 completed both assessments, as did all expert raters. Intrarater reliability for dermatologists was excellent [median (range) ICCs: pustulation, 0.92 (0.75,1); erythema, 0.91 (0.76,1); scaling, 0.87 (0.66,1)], indicating consistent absolute agreement of severity over time, excepting 1 dermatologist who recorded a scaling ICC of 0.66 (95% CI: 0.29, 0.85; ‘good’). Expert raters were within the excellent threshold (ICC ≥0.81). For interrater reliability, absolute agreement among the 26 dermatologists was excellent in all categories and for expert raters in erythema, scaling, and total score; the pustule score ICC was 0.69 (‘good’). The results of the inter- and intrarater assessments demonstrated a high level of reliability for scoring of GPPGA by physicians globally; dermatologists were consistent over time with individual assessments of disease severity.

Competency assessment of residents of Intensive Care Medicine through a simulation-based objective structured clinical evaluation (OSCE). A multicenter observational study

ObjectivesThe current official model of training in Intensive Care Medicine (ICM) in Spain is based on exposure to experiences through clinical rotations. The main objective was to determine the level of competency (I novice to V independent practitioner) achieved by the residents at the end of the 3rd year of training (R3) in ICM through a simulation-based OSCE. Secondary objectives were: (1) To identify gaps in performance, and (2) To investigate the reliability and feasibility of conducting simulation-based assessment at multiple sites. DesignObservational multicenter study. SettingThirteen Spanish ICU Departments. ParticipantsThirty six R3. InterventionThe participants performed on five, 15-min, high-fidelity crisis scenarios in four simulation centers. The performances were video recorded for later scoring by trained raters. Main variables of interestVia a Delphi technique, an independent panel of expert intensivists identified critical essential performance elements (CEPE) for each scenario to define the levels of competency. ResultsA total of 176 performances were analyzed. The internal consistency of the check-lists were adequate (KR-20 range 0.64–0.79). Inter-rater reliability was strong [median Intraclass Correlation Coefficient across scenarios: 0.89 (0.65–0.97)]. Competency levels achieved by R3 were: Level I (18.8%), II (35.2%), III (42.6%), IV/V (3.4%). Overall, a great heterogeneity in performance was observed. ConclusionThe expected level of competency after one year in the ICU was achieved only in half of the performances. A more evidence-based educational approach is needed. Multiple center simulation-based assessment showed feasibility and reliability as an evaluation method of competency. Trial registrationCOBALIDATION. NCT04278976. (https://register.clinicaltrials.gov).

Assessing the robustness of radiomics/deep learning approach in the identification of efficacy of anti-PD-1 treatment in advanced or metastatic non-small cell lung carcinoma patients.

Administration of anti–PD-1 is now a standard therapy in advanced non-small cell lung carcinoma (NSCLC) patients. The clinical application of biomarkers reflecting tumor immune microenvironment is hurdled by the invasiveness of obtaining tissues despite its importance in immunotherapy. This study aimed to develop a robust and non-invasive radiomics/deep learning machine biomarker for predicting the response to immunotherapy in NSCLC patients. Radiomics/deep learning features were exacted from computed tomography (CT) images of NSCLC patients treated with Nivolumab or Pembrolizumab. The robustness of radiomics/deep learning features was assessed against various perturbations, then robust features were selected based on the Intraclass Correlation Coefficient (ICC). Radiomics/deep learning machine-learning classifiers were constructed by combining seven feature exactors, 13 feature selection methods, and 12 classifiers. The optimal model was selected using the mean area under the curve (AUC) and relative standard deviation (RSD). The consistency of image features against various perturbations was high (the range of median ICC: 0.78–0.97), but the consistency was poor in test–retest testing (the range of median ICC: 0.42–0.67). The optimal model, InceptionV3_RELF_Nearest Neighbors classifiers, had the highest prediction efficacy (AUC: 0.96 and RSD: 0.50) for anti–PD-1/PD-L1 treatment. Accuracy (ACC), sensitivity, specificity, precision, and F1 score were 95.24%, 95.00%, 95.50%, 91.67%, and 95.30%, respectively. For successful model robustification, tailoring perturbations for robustness testing to the target dataset is key. Robust radiomics/deep learning features, when paired with machine-learning methodologies, will work on the exactness and the repeatability of anticipating immunotherapy adequacy.

The impact of inter-observer variation in delineation on robustness of radiomics features in non-small cell lung cancer

Artificial intelligence and radiomics have the potential to revolutionise cancer prognostication and personalised treatment. Manual outlining of the tumour volume for extraction of radiomics features (RF) is a subjective process. This study investigates robustness of RF to inter-observer variation (IOV) in contouring in lung cancer. We utilised two public imaging datasets: ‘NSCLC-Radiomics’ and ‘NSCLC-Radiomics-Interobserver1’ (‘Interobserver’). For ‘NSCLC-Radiomics’, we created an additional set of manual contours for 92 patients, and for ‘Interobserver’, there were five manual and five semi-automated contours available for 20 patients. Dice coefficients (DC) were calculated for contours. 1113 RF were extracted including shape, first order and texture features. Intraclass correlation coefficient (ICC) was computed to assess robustness of RF to IOV. Cox regression analysis for overall survival (OS) was performed with a previously published radiomics signature. The median DC ranged from 0.81 (‘NSCLC-Radiomics’) to 0.85 (‘Interobserver’—semi-automated). The median ICC for the ‘NSCLC-Radiomics’, ‘Interobserver’ (manual) and ‘Interobserver’ (semi-automated) were 0.90, 0.88 and 0.93 respectively. The ICC varied by feature type and was lower for first order and gray level co-occurrence matrix (GLCM) features. Shape features had a lower median ICC in the ‘NSCLC-Radiomics’ dataset compared to the ‘Interobserver’ dataset. Survival analysis showed similar separation of curves for three of four RF apart from ‘original_shape_Compactness2’, a feature with low ICC (0.61). The majority of RF are robust to IOV, with first order, GLCM and shape features being the least robust. Semi-automated contouring improves feature stability. Decreased robustness of a feature is significant as it may impact upon the features’ prognostic capability.

Reliability and validity of the Roche PD Mobile Application for remote monitoring of early Parkinson’s disease

Digital health technologies enable remote and therefore frequent measurement of motor signs, potentially providing reliable and valid estimates of motor sign severity and progression in Parkinson’s disease (PD). The Roche PD Mobile Application v2 was developed to measure bradykinesia, bradyphrenia and speech, tremor, gait and balance. It comprises 10 smartphone active tests (with ½ tests administered daily), as well as daily passive monitoring via a smartphone and smartwatch. It was studied in 316 early-stage PD participants who performed daily active tests at home then carried a smartphone and wore a smartwatch throughout the day for passive monitoring (study NCT03100149). Here, we report baseline data. Adherence was excellent (96.29%). All pre-specified sensor features exhibited good-to-excellent test–retest reliability (median intraclass correlation coefficient = 0.9), and correlated with corresponding Movement Disorder Society–Unified Parkinson's Disease Rating Scale items (rho: 0.12–0.71). These findings demonstrate the preliminary reliability and validity of remote at-home quantification of motor sign severity with the Roche PD Mobile Application v2 in individuals with early PD.