Intraclass Correlation Coefficient Estimates Research Articles

Competence of Intensivists in Focused Transthoracic Echocardiography in Intensive Care Unit: A Prospective Observational Study.

Objectives:Focused transthoracic echocardiography (fTTE) in critical care can be used to assess patient's volume status, ventricular contractility, right ventricle chamber size, and valvular abnormalities. The objective of the study was to assess the competency of intensivists in performing fTTE in Intensive Care Unit (ICU) patients after a brief training course by cardiologist using a specific ECHO protocol.Methods:One hundred and four patients in ICU were recruited for this prospective observational study over a period of 12 months. Intensivists were trained for 60 h (2 h/day for 30 days). Intensivists performed fTTE in 82 ICU patients using a specific ECHO protocol developed in consensus with cardiologists. Each patient was assessed by an intensivist and two blinded cardiologists. At the end of the study period, the competency of intensivists was compared with two cardiologists and analyzed using intraclass correlation coefficient (ICC).Results:There were excellent agreement between intensivists and cardiologists in terms of measuring ejection fraction (ICC estimate was 0.973–0.987), valvular function (ICC estimate for mitral valve was 0.940–0.972; ICC estimate for aortic valve was 0.872–0.940), and ICC estimate for pulmonary hypertension was 0.929–0.967. Good reliability has been found for the assessment of volume status with inferior vena cava diameter (ICC estimate for assessing hypovolemia was 0.790–0.902).Conclusion:Intensivists with requisite training in TTE were able to perform focused echocardiography with comparable accuracy to that of cardiologists. Further studies are required to elucidate the therapeutic implications of fTTE performed by the intensivists.

Distribution, variability, and predictors of urinary bisphenol A levels in 50 North Carolina adults over a six-week monitoring period

Bisphenol A (BPA) is commonly manufactured to make polycarbonate plastics and epoxy resins for use in consumer products and packaged goods. BPA has been found in several different types of environmental media (e.g., food, dust, and air). Many cross-sectional studies have frequently detected BPA concentrations in adult urine samples. However, limited data are available on the temporal variability and important predictors of urinary BPA concentrations in adults. In this work, the major objectives were to: 1) quantify BPA levels in duplicate-diet solid food, drinking water, hard floor surface wipe, and urine samples (first-morning void [FMV], bedtime, and 24-h) collected from adults over a six-week monitoring period; 2) determine the temporal variability of urinary BPA levels using concentration, specific gravity (SG) adjusted, creatinine (CR) adjusted, and excretion rate values, and; 3) examine associations between available study factors and urinary BPA concentrations. In 2009–2011, a convenience sample of 50 adults was recruited from residential settings in North Carolina. The participants completed diaries and collected samples during weeks 1, 2, and/or 6 of a six-week monitoring period. BPA was detected in 38%, 4%, and 99% of the solid food (n=775), drinking water (n=50), and surface wipe samples (n=138), respectively. Total BPA (free plus conjugated) was detected in 98% of the 2477 urine samples. Median urinary BPA levels were 2.07ng/mL, 2.20ng/mL-SG, 2.29ng/mg, and 2.31ng/min for concentration, SG-adjusted, CR-adjusted, and excretion rate values, respectively. The intraclass correlation coefficient (ICC) estimates for BPA showed poor reproducibility (≤0.35) for all urine sample types and methods over a day, week, and six weeks. CR-adjusted bedtime voids collected over six-weeks required the fewest, realistic number of samples (n=11) to obtain a reliable biomarker estimate (ICC=0.80). Results of linear mixed-effects models showed that sex, race, season, and CR-level were all significant predictors (p<0.05) of the adults' urinary BPA concentrations. BPA levels in the solid food and surface wipe samples did not contribute significantly to the participants' urinary BPA concentrations. However, a significant positive relationship was observed between solid food intake and urine-based estimates of BPA dose, when aggregated over 24-h periods. Ingestion of BPA via solid food explained only about 20% of the total dose (at the median of the dose distribution), suggesting that these adults were likely exposed to other major unknown (non-dietary) sources of BPA in their everyday environments.

Measurement Under the Microscope: High Variability and Limited Construct Validity in Emergency Department Patient-Experience Scores

We evaluate variability and construct validity in commercially generated patient-experience survey data in a large sample of US emergency departments (EDs). We used Press Ganey patient-experience data from a national emergency medicine group from 2012 to 2015 across 42 facilities and 242 physicians. We estimated variability as month-to-month changes in percentile scores and through intraclass correlations. Construct validity was assessed with linear regression analysis for monthly facility- and physician-level percentile scores. A total of 1,758 facility-months and 10,328 physician-months of data were included. Across facility-months, 40.8% had greater than 10 points of percentile change, 14.7% changed greater than 20 points, and 4.4% changed greater than 30. Across physician-months, 31.9% changed greater than 20 points, 21.5% changed greater than 30, and 13.6% changed greater than 40. Intraclass correlation estimates demonstrated similar variability; however, this was reduced as data were aggregated over fixed time increments. For facility-level construct validity, several facility factors predicted higher scores: teaching status; more older, male, and discharged patients without Medicaid insurance; lower patient volume; less requirement for physician night coverage; and shorter lengths of stay for discharged patients. For physician-level construct validity, younger physician age, participating in satisfaction training, increasing relative value units per visit, more commercially insured patients, higher computed tomography or magnetic resonance imaging use, working during less crowded times, and fewer night shifts predicted higher scores. In this sample, both physician- and facility-level patient-experience data varied greatly month to month, with physician variability being considerably higher. Facility-level scores have greater construct validity than physician-level ones. Optimizing data gathering may reduce variability in ED patient-experience data and better inform decisionmaking, quality measurement, and pay for performance.

Intraclass correlation: Improved modeling approaches and applications for neuroimaging.

Intraclass correlation (ICC) is a reliability metric that gauges similarity when, for example, entities are measured under similar, or even the same, well-controlled conditions, which in MRI applications include runs/sessions, twins, parent/child, scanners, sites, and so on. The popular definitions and interpretations of ICC are usually framed statistically under the conventional ANOVA platform. Here, we provide a comprehensive overview of ICC analysis in its prior usage in neuroimaging, and we show that the standard ANOVA framework is often limited, rigid, and inflexible in modeling capabilities. These intrinsic limitations motivate several improvements. Specifically, we start with the conventional ICC model under the ANOVA platform, and extend it along two dimensions: first, fixing the failure in ICC estimation when negative values occur under degenerative circumstance, and second, incorporating precision information of effect estimates into the ICC model. These endeavors lead to four modeling strategies: linear mixed-effects (LME), regularized mixed-effects (RME), multilevel mixed-effects (MME), and regularized multilevel mixed-effects (RMME). Compared to ANOVA, each of these four models directly provides estimates for fixed effects and their statistical significances, in addition to the ICC estimate. These new modeling approaches can also accommodate missing data and fixed effects for confounding variables. More importantly, we show that the MME and RMME approaches offer more accurate characterization and decomposition among the variance components, leading to more robust ICC computation. Based on these theoretical considerations and model performance comparisons with a real experimental dataset, we offer the following general-purpose recommendations. First, ICC estimation through MME or RMME is preferable when precision information (i.e., weights that more accurately allocate the variances in the data) is available for the effect estimate; when precision information is unavailable, ICC estimation through LME or the RME is the preferred option. Second, even though the absolute agreement version, ICC(2,1), is presently more popular in the field, the consistency version, ICC(3,1), is a practical and informative choice for whole-brain ICC analysis that achieves a well-balanced compromise when all potential fixed effects are accounted for. Third, approaches for clear, meaningful, and useful result reporting in ICC analysis are discussed. All models, ICC formulations, and related statistical testing methods have been implemented in an open source program 3dICC, which is publicly available as part of the AFNI suite. Even though our work here focuses on the whole-brain level, the modeling strategy and recommendations can be equivalently applied to other situations such as voxel, region, and network levels.

Test-retest reliability for pulse wave velocity and cardio-ankle vascular index among Afro-Caribbean young adults

BackgroundThis study evaluated the test-retest reliability of carotid-femoral pulse wave velocity (PWV) and cardio-ankle vascular index (CAVI) among young adults in Jamaica.MethodsWe recruited participants from the Jamaica 1986 Birth Cohort Study. PWV was measured using the Arteriograph device™ (TensioMed, Budapest) and CAVI with the VaSera™ device (Fukuda Denshi, Tokyo). Both measurements were done twice on the same day with a 1-hour interval between measurements. Test-retest reliability was estimated using the intra-class correlation coefficient (ICC) and Bland-Altman plots. Kappa statistic was used to assess agreement between repeated tests in classifying participants as high PWV or CAVI, defined as being in the upper tertile of measurements.ResultsAnalyses included 89 participants (43 males; 46 females; mean age 28.4 ± 0.50 years). Mean PWV for first and second readings were 6.56 cm/s and 6.64 cm/s, respectively (mean difference −0.08 [95%CI −0.18, 0.03, p = 0.142]). Mean values for first and second CAVI were 6.53 and 6.20, respectively, (mean difference 0.34 [95%CI 0.18, 0.50, p < 0.001]). ICC for PWV was 0.88 (95%CI 0.83, 0.92) and for CAVI 0.57 (95%CI 0.41, 0.69). Bland-Altman plots indicated that measurements taken from both devices were highly reproducible, with most points (85/89 for PWV; 86/89 for CAVI) falling within 2 SD of the mean difference. Kappa statistic was 0.76 for PWV and 0.56 for CAVI.ConclusionPWV (Arteriograph™) and CAVI (VaSera™) have good test-retest reliability among Jamaican youth adults; however repeated CAVI values were marginally lower than the first measurement and the ICC and kappa estimates were lower.

Diurnal variation and repeatability of arterial stiffness and cardiac output measurements in the third trimester of uncomplicated pregnancy.

To investigate same day repeated measures and diurnal variation of arterial stiffness, cardiac output (CO), stroke volume (SV) and total peripheral resistance (TPR) during the third trimester of normal pregnancy. Pulse wave velocity (PWV) and augmentation index (AIx) were recorded using the Arteriograph, while CO, SV and TPR were recorded using noninvasive cardiac output monitoring. The measurements were obtained in the third trimester of pregnancy from 21 healthy pregnant women at four time points (morning, afternoon, evening and midnight) over a 24-h period. Triplicate measurements of 67 women were obtained at 5-min intervals to assess repeatability between measurements within a patient. Diurnal measurements of arterial stiffness for brachial AIx, aortic AIx and PWV were not statistically significantly different at any of the four time points. Estimated means (SD) for PWV at the four stated time points were 7.81 (2.05), 8.45 (1.68), 7.87 (1.74) and 7.64 m/s (1.15), respectively (P = 0.267). Estimates for AIx at those time points were 10.22 (15.62), 4.44 (10.07), 6.49 (10.92) and 8.40% (8.16), respectively (P = 0.295). Similarly, mean arterial pressure, SV, SV index and TPR did not show any evidence of diurnal variation. However, we observed that the mean CO, cardiac index (CI) and heart rate (HR) varied from morning to midnight; the mean CO, HR and CI increased significantly in the afternoon compared with the corresponding mean morning measurements in a similar fashion to HR. Mean (SD) CO estimates at the four stated time points were 5.90 (1.33), 6.38 (1.49), 6.18 (1.43) and 5.80 ml/min (1.19), respectively, (P < 0.001), whereas mean CI estimates were 3.65 (0.58), 3.93 (0.68), 3.81 (0.65), and 3.57 (0.48), respectively, (P < 0.001), and mean HR estimates were 95 (12), 98 (13), 95 (12) and 88 (12.98), respectively (P < 0.001). Triplicate measurements of 61 women in our repeatability study showed moderate-to-high correlation between observations on the same woman for all Arteriograph and noninvasive cardiac output monitoring variables (estimates of intraclass correlation ranged from 0.49 to 0.91). With the exception of CO, CI and HR which showed a diurnal variation, measurements of most haemodynamic parameters did not change significantly from morning to midnight, suggesting there was no evidence of systematic differences in the mean values of these variables at these time points. Multiple consecutive noninvasive measurements of vascular stiffness, CO, SV and TPR were highly correlated confirming repeatability of measurements in the third trimester of uncomplicated pregnancy, so these haemodynamic measurements do not need to be undertaken at a specific time period of the day.

Reproducibility of diaphragm thickness measurements by ultrasonography in patients on mechanical ventilation

AIMTo prospectively evaluate the reproducibility of diaphragm thickness measurement by ultrasonography at the bedside by critical care physicians in patients on invasive mechanical ventilation.METHODSIn a prospective observational study of 64 invasively ventilated patients, diaphragmatic thickness measurement was taken by 2 different observers at the same site. Three measurements were taken by each observer and averaged. The intraobserver and interobserver variability was assessed by estimation of intraclass correlation coefficient. The limits of agreement were plotted as the difference between two observations against the average of the two observations in Bland and Altman analysis.RESULTSThe mean diaphragm thickness at the functional residual capacity was 2.29 ± 0.4 mm and the lower limit of the normal, i.e., the 5th percentile was 1.7 mm (95%CI: 1.6-1.8). The intraclass correlation coefficient for intraobserver variability was 0.986 (95%CI: 0.979-0.991) with a P value of < 0.001. The intraclass correlation coefficient for interobserver variability was 0.987 (95%CI: 0.949-0.997) with a P value of < 0.001. In Bland and Altman analysis, both intraobserver and interobserver measurements showed high limits of agreement.CONCLUSIONOur study demonstrates that the measurement of diaphragm thickness by ultrasound can be accurately performed by critical care physicians with high degree of reproducibility in patients on mechanical ventilation.

Validation of a Smartphone App for the Assessment of Sedentary and Active Behaviors.

BackgroundAlthough current technological advancements have allowed for objective measurements of sedentary behavior via accelerometers, these devices do not provide the contextual information needed to identify targets for behavioral interventions and generate public health guidelines to reduce sedentary behavior. Thus, self-reports still remain an important method of measurement for physical activity and sedentary behaviors.ObjectiveThis study evaluated the reliability, validity, and sensitivity to change of a smartphone app in assessing sitting, light-intensity physical activity (LPA), and moderate-vigorous physical activity (MVPA).MethodsAdults (N=28; 49.0 years old, standard deviation [SD] 8.9; 85% men; 73% Caucasian; body mass index=35.0, SD 8.3 kg/m2) reported their sitting, LPA, and MVPA over an 11-week behavioral intervention. During three separate 7-day periods, participants wore the activPAL3c accelerometer/inclinometer as a criterion measure. Intraclass correlation (ICC; 95% CI) and bias estimates (mean difference [δ] and root of mean square error [RMSE]) were used to compare app-based reported behaviors to measured sitting time (lying/seated position), LPA (standing or stepping at <100 steps/minute), and MVPA (stepping at >100 steps/minute).ResultsTest-retest results suggested moderate agreement with the criterion for sedentary time, LPA, and MVPA (ICC=0.65 [0.43-0.82], 0.67 [0.44-0.83] and 0.69 [0.48-0.84], respectively). The agreement between the two measures was poor (ICC=0.05-0.40). The app underestimated sedentary time (δ=-45.9 [-67.6, -24.2] minutes/day, RMSE=201.6) and overestimated LPA and MVPA (δ=18.8 [-1.30 to 38.9] minutes/day, RMSE=183; and δ=29.3 [25.3 to 33.2] minutes/day, RMSE=71.6, respectively). The app underestimated change in time spent during LPA and MVPA but overestimated change in sedentary time. Both measures showed similar directions in changed scores on sedentary time and LPA.ConclusionsDespite its inaccuracy, the app may be useful as a self-monitoring tool in the context of a behavioral intervention. Future research may help to clarify reasons for under- or over-reporting of behaviors.

The development of a magnetic resonance imaging index for fistulising Crohn's disease.

Magnetic resonance imaging (MRI) is the gold standard for assessment of perianal fistulising Crohn's disease (CD). The Van Assche index is the most commonly used MRI fistula index. To assess the reliability of the Van Assche index, and to modify the instrument to improve reliability and create a novel index for fistulising CD. A consensus process developed scoring conventions for existing Van Assche index component items and new items. Four experienced radiologists evaluated 50 MRI images in random order on three occasions. Reliability was assessed by estimates of intraclass correlation coefficients (ICCs). Common sources of disagreement were identified and recommendations made to minimise disagreement. A mixed effects model used a 100mm visual anologue scale (VAS) for global severity as outcome and component items as predictors to create a modified Van Assche index. Intraclass correlation coefficients (95% confidence intervals) for intra-rater reliability of the original and modified Van Assche indices and the VAS were 0.86 (0.81-0.90), 0.90 (0.86-0.93) and 0.86 (0.82-0.89). Corresponding ICCs for inter-rater reliability were 0.66 (0.52-0.76), 0.67 (0.55-0.75) and 0.58 (0.47-0.66). Sources of disagreement included number, location, and extension of fistula tracts, and rectal wall involvement. A modified Van Assche index (range 0-24) was created that included seven component items. Although "almost perfect" intra-rater reliability was observed for the assessment of MRI images for fistulising CD using the Van Assche index, inter-rater reliability was considerably lower. Our modification of this index should result in a more optimal instrument.

P213 Agreement between Crohn's disease endoscopy severity scores derived from live local, delayed local video-recorded and central readings

Background: The Crohn's Disease Endoscopic Index of Severity (CDEIS) and the Simple Endoscopic Score for Crohn's Disease (SES-CD) are used to assess endoscopic disease severity in Crohn's disease (CD). Central reading has become the standard practice in clinical trials for inclusion and evaluation of endoscopic response to therapy. However, the agreement between severity scores (CDEIS or SES-CD) derived from live local, delayed local video-recorded and central readings has not been studied. Methods: We conducted a monocentric prospective study between April 2015 and December 2015. Fifty-three CD patients were recruited by three endoscopists trained to endoscopic readings for score calculation (TU, JM and BC). All colonoscopies were recorded on-site and videos were labeled according to the segment of interest: ileum, right colon, transverse colon, left and sigmoid colon, and rectum. Each endoscopist performed immediate readings of his patients and delayed local video-recorded readings of his patients' colonoscopies at least 3 months after the live local reading. Two central readers (JCD and FP) read all videos. CDEIS and SES-CD scores were then computed automatically based on all readings. Intra-class correlation coefficients (ICC) were estimated and Bland and Altman plots were used to assess the agreement between severity scores derived from various readings. In a first step, we studied the agreement between severity scores derived from live local (L) and central (C1 and C2) readings and L and delayed local video-recorded (D) readings. Results: ICC estimates of CDEIS (n=44) and SES-CD (n=46) scores were respectively 0.89; 95% confidence interval (CI), 0.80–0.94 and 0.94; 95% CI, 0.87–0.97 between L and C1 and 0.80; 95% CI, 0.58–0.90 and 0.85; 95% CI, 0.74–0.91 between L and C2. ICC estimates between L and D were respectively 0.87; 95% CI, 0.78–0.93 and 0.88; 95% CI, 0.79–0.93. An example of Bland and Altman plots derived from L and C1 are shown in figure 1 for CDEIS and figure 2 for SES-CD scores. Figure 1. Bland and Altman plots of CDEIS derived from live local reading and central reading 1. Figure 2. Bland and Altman plots of SES-CD derived from live local reading and central reading 1. Conclusions: Overall, the agreement between severity scores (CDEIS or SES-CD) derived from live local, delayed local video-recorded and central readings is excellent. This data suggests that both local readings by trained endoscopists and central reading could be used in clinical trials.

Beta-binomial model for meta-analysis of odds ratios.

In meta‐analysis of odds ratios (ORs), heterogeneity between the studies is usually modelled via the additive random effects model (REM). An alternative, multiplicative REM for ORs uses overdispersion. The multiplicative factor in this overdispersion model (ODM) can be interpreted as an intra‐class correlation (ICC) parameter. This model naturally arises when the probabilities of an event in one or both arms of a comparative study are themselves beta‐distributed, resulting in beta‐binomial distributions. We propose two new estimators of the ICC for meta‐analysis in this setting. One is based on the inverted Breslow‐Day test, and the other on the improved gamma approximation by Kulinskaya and Dollinger (2015, p. 26) to the distribution of Cochran's Q. The performance of these and several other estimators of ICC on bias and coverage is studied by simulation. Additionally, the Mantel‐Haenszel approach to estimation of ORs is extended to the beta‐binomial model, and we study performance of various ICC estimators when used in the Mantel‐Haenszel or the inverse‐variance method to combine ORs in meta‐analysis. The results of the simulations show that the improved gamma‐based estimator of ICC is superior for small sample sizes, and the Breslow‐Day‐based estimator is the best for n⩾100. The Mantel‐Haenszel‐based estimator of OR is very biased and is not recommended. The inverse‐variance approach is also somewhat biased for ORs≠1, but this bias is not very large in practical settings. Developed methods and R programs, provided in the Web Appendix, make the beta‐binomial model a feasible alternative to the standard REM for meta‐analysis of ORs. © 2017 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.

Psychometric properties of the Swedish version of the Treatment Outcome Satisfaction Questionnaire

PurposePatient satisfaction is an outcome measure for low-back pain (LBP) interventions which allows clinicians to design patient-oriented treatments. The Treatment Outcome Satisfaction Questionnaire (TOSQ) is an English instrument constructed for such evaluations, and no equivalent instruments exist for the Swedish population. This study, therefore, translated TOSQ into Swedish and assessed the translated version’s psychometric properties for patients with LBP.MethodsA cross-cultural adaptation was used to translate TOSQ into Swedish. Subsequently, data from 131 patients with LBP whom undergone physiotherapy were consecutively aggregated and analyzed in a Rasch rating scale model with person measures standardized at 0–100 logits to evaluate the translated scale’s validity. Finally, test–retest reliability of the Swedish version of TOSQ (TOSQ-S) was quantified via an intraclass correlation coefficient (ICC) and the standard error of measurement (SEM) in 41 patients.ResultsTOSQ was successfully translated into Swedish; however, while some Rasch model indices supported the translated scale’s unidimensionality, one out of eight items and 12 out of 131 subjects misfitted the model. Scale optimization resulted in a 6-item subconfiguration, for which all items fitted the model, person misfits were reduced to ten subjects, and the person separation index increased from 1.86 to 2.04. ICC and SEM estimates suggested acceptable reliability for the six-item TOSQ-S at 0.66 and 6.6 logits, respectively.ConclusionsA six-item TOSQ-S configuration showed acceptable psychometric properties and is suitable for measuring treatment outcome satisfaction of physiotherapy in patients with LBP.

Designing Large-Scale Multisite and Cluster-Randomized Studies of Professional Development

ABSTRACTWe develop a theoretical and empirical basis for the design of teacher professional development studies. We build on previous work by (a) developing estimates of intraclass correlation coefficients for teacher outcomes using two- and three-level data structures, (b) developing estimates of the variance explained by covariates, and (c) modifying the conventional optimal design framework to include differential covariate costs so as to capture the point at which the cost of collecting a covariate overtakes the reduction in variance it supplies. We illustrate the use of these estimates to explore the absolute and relative sensitivity of multilevel designs in teacher professional development studies. The results from these analyses are intended to guide researchers in making more-informed decisions about the tradeoffs and considerations involved in selecting study designs for assessing the impacts of professional development programs.

Confidence intervals for intraclass correlation coefficients in variance components models

Confidence intervals for intraclass correlation coefficients in agreement studies with continuous outcomes are model-specific and no generic approach exists. This paper provides two generic approaches for intraclass correlation coefficients of the form [Formula: see text] The first approach uses Satterthwaite's approximation and an F-distribution. The second approach uses the first and second moments of the intraclass correlation coefficient estimate in combination with a Beta distribution. Both approaches are based on the restricted maximum likelihood estimates for the variance components involved. Simulation studies are conducted to examine the coverage probabilities of the confidence intervals for agreement studies with a mix of small sample sizes. Two different three-way variance components models and balanced and unbalanced one-way random effects models are investigated. The proposed approaches are compared with other approaches developed for these specific models. The approach based on the F-distribution provides acceptable coverage probabilities, but the approach based on the Beta distribution results in accurate coverages for most settings in both balanced and unbalanced designs. A real agreement study is provided to illustrate the approaches.

Estimates of Intraclass Correlation Coefficients and Other Design Parameters for Studies of School-Based Nutritional Interventions.

When cluster randomized trials are used to evaluate school-based nutritional interventions such as school lunch programs, design-stage estimates of the required sample size must take into account the correlation in outcomes among individuals within each cluster (e.g., classrooms, schools, or districts). Estimates of the necessary parameters have been carefully developed for educational interventions, but for nutritional interventions the literature is thin. Using data from two large multi-school, multi-district impact evaluations conducted in the United States, this article calculates estimates of the design parameters required for sizing school-based nutritional studies. The large size of the trials (252 and 1,327 schools) yields precise estimates of the parameters of interest. Variance components are estimated by fitting random-intercept multilevel models in Stata. School-level intraclass correlations are similar to those typically found for educational outcomes. In particular, school-level estimates range from less than .01 to .26 across the two studies, and district-level estimates ranged from less than .01 to .19. This suggests that cluster randomized trials of nutritional interventions may require samples with numbers of schools similar to the education studies to detect similar effect sizes.

Comparing Between- and Within-Group Variances in a Two-Level Study: A Latent Variable Modeling Approach to Evaluating Their Relationship.

This note is concerned with examining the relationship between within-group and between-group variances in two-level nested designs. A latent variable modeling approach is outlined that permits point and interval estimation of their ratio and allows their comparison in a multilevel study. The procedure can also be used to test various hypotheses about the discrepancy between these two variances and assist with their relationship interpretability in empirical investigations. The method can also be utilized as an addendum to point and interval estimation of the popular intraclass correlation coefficient in hierarchical designs. The discussed approach is illustrated with a numerical example.

Finding Efficiency in the Design of Large Multisite Evaluations

Little is known empirically about intraclass correlations (ICCs) for multisite cluster randomized trial (MSCRT) designs, particularly in science education. In this study, ICCs suitable for science achievement studies using a three-level (students in schools in districts) MSCRT design that block on district are estimated and examined. Estimates of ICCs, by district, are computed for unconditional two-level (students in schools) hierarchical linear models using student-level science achievement raw scores as measured by the Texas Assessment of Knowledge and Skills in 2010–2011 for Grades 5, 8, 10, and 11. The average within-district ICCs for MSCRTs of science achievement range from 0.0781 to 0.0982 across grades. For Grade 5, a significant difference is found in the ICC estimate suitable for MSCRTs with many districts versus only a few districts. Consequently, evaluators of science education interventions should exhibit care when selecting ICC estimates for three-level MSCRT designs that block on district.

Simulator and 2 tools: Validation of performance measures from a novel neurosurgery simulation model using the current Standards framework

Ventriculostomy is a common neurosurgical procedure with a relatively steep learning curve. A low-cost, high-fidelity simulator paired with procedure-specific performance measures would provide a safe environment to teach ventriculostomy procedural skills. The same validated simulation model could also allow for assessment of trainees' proficiencies with measures that align with Accreditation Council for Graduate Medical Education milestones. This study extends previous work to evaluate validity evidence from the simulator, its newly developed performance assessment, the VentricolostomyProcedural Assessment Tool, and the Objective Structured Assessment for Technical Skills. After Institutional Review Board exemption, performance data were collected from 11 novice and 3 expert neurosurgeons (n=14). Participants self-reported their ability to perform tasks on the simulator using the Ventricolostomy Procedural Assessment Tool, an 11-item, step-wise instrument with 5-point rating scales ranging from 1 (unable to perform) to 5 (performs easily and smoothly). De-identified operative performances were videotaped and independently rated by 3 neurosurgeons, using the Ventricolostomy Procedural Assessment Tool and Objective Structured Assessment for Technical Skills. We evaluated multiple sources of validity evidence (2014 Standards) to examine psychometric quality of the measures and to test our assumption that the tools could discriminate between novice and expert performances adequately. We used a multifacet Rasch model and traditional indices, such as Cronbach alpha, intraclass correlation, and Wilcoxon signed-rank test estimates. Validity evidence relevant to test content and response processes was supported adequately. Evidence of internal structure was supported by high interitem consistency (n=0.95) and inter-rater agreement for most Ventricolostomy Procedural Assessment Tool items (Intraclass correlation coefficient=[0.00, 0.91]) and all Objective Structured Assessment for Technical Skills items (Intraclass correlation coefficient=[0.80, 0.93]). Overall, novices performed at a lower level than experts on both scales (P<.05), supporting evidence relevant to relationships to other variables. Deeper analysis of novice/expert ratings indicated novices attained lower performances ratings for all Ventricolostomy Procedural Assessment Tool and Objective Structured Assessment for Technical Skills items, but statistical significance was only achieved for the Objective Structured Assessment for Technical Skills items (P<.01). Rater bias estimates were favorable, supporting evidence relevant to consequences of testing. Despite a small sample, favorable evidence using current Standards supports the use of the novel simulator and both tools combined for skills training and performance assessment, but challenges (potential threats to validity) should be considered prior to implementation.

Accuracy Of A Smartphone-based “App” For The Assessement Of Sedentary And More Active Behaviors

Mobile devices provide a convenient platform for log-based assessments of sedentary and physical activity (PA) behaviors. PURPOSE: This study aimed to evaluate the accuracy of a smartphone-based “app” designed to assess context-specific forms of sitting, light-intensity PA (LPA), and moderate-vigorous PA (MVPA). METHODS: Adults (N=23 [323 observations]; 49.0±8.9 years; 85% men; 73% Caucasian; BMI=35.0±8.3kg/m2) reported their sitting, LPA, and MVPA over the course of an 11-week behavioral intervention. During three separate 7d time period, participants simultaneously wore the ActivPAL3c™ accelerometry/inclinometer as a criterion measure. Intraclass correlation (ICC [95% CI]) and bias estimates (mean difference [¯d] and absolute percent error [APE = difference/direct observation x 100]) were used to compare app-based reported behaviors to measure sedentary time (lying/seated position), LPA (standing or stepping < 3 metabolic equivalents [METs], and MVPA (stepping ≥ 3METs). RESULTS: Sitting time accuracy was moderate (ICC=0.50 [0.37, 0.60]) with under-reporting (¯d = -127 [-612.41, -356.73] minutes/day; APE = -8.61 [-155.73, 140.51]). LPA and MVPA time accuracy was also moderate (ICC=0.64 [0.55, .72] and 0.55 [0.32, 0.71], respectively) with over-reporting (¯d = 293.00 [-39.06, 625.06] minutes/day; APE = 31.87 [-138.28, 202.02]) and (¯d = 19.46 [-133.20, 172.12] minutes/day; APE=83.93 [-387.21, -555.07]). CONCLUSION: The smartphone-based log targeting sedentary and more active behaviors underreported sedentary time and overreported LPA and MVPA with only moderate accuracy relative to an objective monitor. This tool may still be useful as a self-monitoring tool in the context of a behavioral intervention as well as a tool to capture context-specific forms of sedentary, LPA, and MVPA behaviors. Given the wide variations in accuracy, future research may explain some reasons for under- or over-reporting of behaviors.

Inter- and Intra-Observer Repeatability of Quantitative Whole-Body, Diffusion-Weighted Imaging (WBDWI) in Metastatic Bone Disease.

Quantitative whole-body diffusion-weighted MRI (WB-DWI) is now possible using semi-automatic segmentation techniques. The method enables whole-body estimates of global Apparent Diffusion Coefficient (gADC) and total Diffusion Volume (tDV), both of which have demonstrated considerable utility for assessing treatment response in patients with bone metastases from primary prostate and breast cancers. Here we investigate the agreement (inter-observer repeatability) between two radiologists in their definition of Volumes Of Interest (VOIs) and subsequent assessment of tDV and gADC on an exploratory patient cohort of nine. Furthermore, each radiologist was asked to repeat his or her measurements on the same patient data sets one month later to identify the intra-observer repeatability of the technique. Using a Markov Chain Monte Carlo (MCMC) estimation method provided full posterior probabilities of repeatability measures along with maximum a-posteriori values and 95% confidence intervals. Our estimates of the inter-observer Intraclass Correlation Coefficient (ICCinter) for log-tDV and median gADC were 1.00 (0.97–1.00) and 0.99 (0.89–0.99) respectively, indicating excellent observer agreement for these metrics. Mean gADC values were found to have ICCinter = 0.97 (0.81–0.99) indicating a slight sensitivity to outliers in the derived distributions of gADC. Of the higher order gADC statistics, skewness was demonstrated to have good inter-user agreement with ICCinter = 0.99 (0.86–1.00), whereas gADC variance and kurtosis performed relatively poorly: 0.89 (0.39–0.97) and 0.96 (0.69–0.99) respectively. Estimates of intra-observer repeatability (ICCintra) demonstrated similar results: 0.99 (0.95–1.00) for log-tDV, 0.98 (0.89–0.99) and 0.97 (0.83–0.99) for median and mean gADC respectively, 0.64 (0.25–0.88) for gADC variance, 0.85 (0.57–0.95) for gADC skewness and 0.85 (0.57–0.95) for gADC kurtosis. Further investigation of two anomalous patient cases revealed that a very small proportion of voxels with outlying gADC values lead to instability in higher order gADC statistics. We therefore conclude that estimates of median/mean gADC and tumour volume demonstrate excellent inter- and intra-observer repeatability whilst higher order statistics of gADC should be used with caution when ascribing significance to clinical changes.