Intersite Consistency Research Articles

Evaluating Intersite Consistency Across 11 Geographically Distinct Pediatric Clerkship Training Sites: Providing Assurance That Educational Comparability Is Possible

We compared core pediatric clerkship student assessments across 11 geographically distinct learning environments following a major curriculum change. We sought to determine if intersite consistency existed, which can be used as a marker of program evaluation success.We evaluated students' overall pediatric clerkship performance along with individual assessments that target our clerkship learning objectives. Using the data of graduating classes from 2015 to 2019 (N = 859), we conducted an analysis of covariance and multivariate logistic regression analysis to investigate whether the performance varied across training sites.Of the students, 833 (97%) were included in the study. The majority of the training sites did not show statistically significant differences from each other. After controlling for the Medical College Admission Test total score and the average pre-clerkship National Board of Medical Examiners final exam score, the clerkship site only explained a 3% additional variance of the clerkship final grade.Over the ensuing 5-year period after a curriculum overhaul to an 18-month, integrated module pre-clerkship curriculum, we found that student pediatric clerkship performance in clinical knowledge and skills did not differ significantly across 11 varied geographic teaching sites when controlling for students' pre-clerkship achievement. Specialty-specific curriculum resources, faculty development tools, and assessment of learning objectives may provide a framework for maintaining intersite consistency when faced with an expanding network of teaching facilities and faculty.

Eosinophilic oesophagitis endotype classification by molecular, clinical, and histopathological analyses: a cross-sectional study

Eosinophilic oesophagitis is understood in terms of quantifiable histological, endoscopic, and molecular features. Data are scant for inter-relations of these features and their potential to identify distinct disease endotypes. We aimed to identify clinical-pathological correlations between endoscopic and histological disease variables by transcription profiling of the oesophagus of patients with eosinophilic oesophagitis of varying severity and disease activity states. We did a cross-sectional study across ten hospital sites in the USA associated with the Consortium of Eosinophilic Gastrointestinal Disease Researchers. We analysed oesophageal biopsy specimens taken from paediatric and adult patients with eosinophilic oesophagitis (discovery cohort), using the eosinophilic oesophagitis diagnostic panel (EDP), a set of 96 informative transcripts. Histological and endoscopic features were assessed by quantification of oesophageal eosinophils and use of the eosinophilic oesophagitis histology scoring system (HSS) and the eosinophilic oesophagitis endoscopic reference score (EREFS). Associations among the various histological, endoscopic, and molecular features were analysed by Spearman correlation. Results were replicated in a biologically independent, single-centre, validation cohort of patients with active eosinophilic oesophagitis. The discovery cohort contained 185 samples and the validation cohort comprised 100 specimens. In the discovery cohort, EDP showed intersite consistency, significant correlation with oesophageal eosinophils (p<0·0001), and similar findings between paediatric and adult patients. Of eight HSS domains, basal zone hyperplasia correlated with the EDP (median Spearman ρ 0·47 [IQR 0·36-0·60]). Of five EREFS features, distal furrows correlated with the EDP (median Spearman ρ 0·42 [0·32-0·50]). By analysing active eosinophilic oesophagitis in the discovery cohort, the EDP identified three clusters associated with distinct endotypes (termed EoEe1-3) despite similar eosinophil levels. EoEe1 was associated with a normal-appearing oesophagus (risk ratio [RR] 3·27, 95% CI 1·04-10·27; p=0·0443), an inverse association with a history of oesophageal dilation (0·27, 0·09-0·82; p=0·0105) and showed relatively mild histological, endoscopic, and molecular changes. EoEe2 showed an inflammatory and steroid-refractory phenotype (RR 2·77, 95% CI 1·11-6·95; p=0·0376) and had the highest expression of inflammatory cytokines and steroid-responding genes. EoEe3 was associated with a narrow-calibre oesophagus (RR 7·98, 95% CI 1·84-34·64; p=0·0013) and adult onset (2·22, 1·19-4·12; p=0·0155), and showed the highest degree of endoscopic and histological severity and the lowest expression of epithelial differentiation genes. These endotypes were replicated in the validation cohort by clustering and with an eosinophilic oesophagitis endotype-prediction algorithm. Our new disease classification stratifies patients with eosinophilic oesophagitis into subgroups with potential clinical and therapeutic significance and provides a framework for a precision medicine approach to eosinophilic oesophagitis. National Institutes of Health.

Reconciling oxygen isotope sclerochronology with interpretations of millennia of seasonal shellfish collection on the Pacific Northwest Coast

The results of high-resolution stable oxygen isotope (δ18O) sclerochronology of 139 butter clam (Saxidomus gigantea) shells from nine archaeological shell midden sites on the central coast of British Columbia, Canada demonstrate clear patterns of multi-season harvest at most sites and specific seasonal harvest at particular site locations. Although the results are based on small numbers of shells relative to an unknown number of seasonal clam harvesting events over the centuries or millennia of site occupation, they can be interpreted with confidence based on regional inter-site consistency, fit with a broader range of ancillary archaeological evidence, and the relative stability of seasonal patterns through iterative increases in sample size. On the basis of these results we outline a pragmatic approach to the selection of shells for seasonality analysis and for the interpretation of seasonal harvest patterns. This approach depends neither on the indeterminate reliability of probabilistic sampling of midden deposits and constituents or the prohibitively costly and still potentially unreliable use of large shell samples drawn from deposits distributed across sites.

Longitudinal reproducibility of default-mode network connectivity in healthy elderly participants: A multicentric resting-state fMRI study

To date, limited data are available regarding the inter-site consistency of test–retest reproducibility of functional connectivity measurements, in particular with regard to integrity of the Default Mode Network (DMN) in elderly participants. We implemented a harmonized resting-state fMRI protocol on 13 clinical scanners at 3.0T using vendor-provided sequences. Each site scanned a group of 5 healthy elderly participants twice, at least a week apart. We evaluated inter-site differences and test–retest reproducibility of both temporal signal-to-noise ratio (tSNR) and functional connectivity measurements derived from: i) seed-based analysis (SBA) with seed in the posterior cingulate cortex (PCC), ii) group independent component analysis (ICA) separately for each site (site ICA), and iii) consortium ICA, with group ICA across the whole consortium. Despite protocol harmonization, significant and quantitatively important inter-site differences remained in the tSNR of resting-state fMRI data; these were plausibly driven by hardware and pulse sequence differences across scanners which could not be harmonized. Nevertheless, the tSNR test–retest reproducibility in the consortium was high (ICC=0.81). The DMN was consistently extracted across all sites and analysis methods. While significant inter-site differences in connectivity scores were found, there were no differences in the associated test–retest error. Overall, ICA measurements were more reliable than PCC-SBA, with site ICA showing higher reproducibility than consortium ICA. Across the DMN nodes, the PCC yielded the most reliable measurements (≈4% test–retest error, ICC=0.85), the medial frontal cortex the least reliable (≈12%, ICC=0.82) and the lateral parietal cortices were in between (site ICA). Altogether these findings support usage of harmonized multisite studies of resting-state functional connectivity to characterize longitudinal effects in studies that assess disease progression and treatment response.

Experimentation Standards for Crisis Informatics

Easy access to online media has led to an escalation of researchers and companies developing information analysis systems for use in crisis response. These systems use state of the art text mining algorithms to extract information from online media to support crisis responders and other users. Published research in this area has primarily focused on systems leveraging public social media feeds such as Twitter. These results often are supported by samples of social media manually analyzed using a combination of hired assessors and data mining algorithms. Unfortunately, despite leveraging open data like Twitter, the current experimental environment for crisis informatics does not rely on open test collections or evaluation methodologies. Social media samples are often collected by individual research sites and not shared. Assessment is often conducted by crowd-workers or in-house editors, at times with little attention paid to inter-site consistency or to real use cases. Consequently, crisis informatics results are difficult to compare across research publications and to apply to real situations. This article makes the argument that information access algorithms that support crisis response require standard evaluation metrics and experimental corpora. There are two objectives for such a initiative: (a) encourage open and reproducible experimentation on crisis informatics, and (b) develop standard evaluation metrics for use by decision-makers. Information retrieval researchers are in a unique position to assist in the development of algorithms, methodology, and working systems for use in crisis response.

Retrieval of tropospheric column-averaged CH&amp;lt;sub&amp;gt;4&amp;lt;/sub&amp;gt; mole fraction by solar absorption FTIR-spectrometry using N&amp;lt;sub&amp;gt;2&amp;lt;/sub&amp;gt;O as a proxy

Abstract. Tropospheric column-averaged CH4 mole fractions were derived from ground-based column absorption measurements. The method uses stratospheric N2O columns to correct for the stratospheric contribution to the CH4 total column. The method was applied to four Total Carbon Column Observing Network (TCCON) sites covering locations from the Northern Arctic to the tropics. It performs well for all sites. The derived tropospheric CH4 concentrations were compared with profiles measured by aircraft at three sites. The results indicate an inter-site consistency within 6 ppb (~0.3%). With aircraft profiles up to 3 km, the seasonal behavior of the derived tropospheric CH4 concentration was also checked, revealing a difference of around 20 ppb. The mean relative uncertainty of the four sites, as estimated from the daily standard deviations, is 0.23%.

Inter-Site Consistency at a Multi-Site Psychiatry Clerkship

This study examines the effects of clinical site assignment within a multiple-site psychiatry clerkship program on the convergent outcome of the National Board of Medical Examiners (NBME) subject examination. NBME scores, controlled for baseline pre-clerkship knowledge base as measured by second year human behavior scores, were compared for inpatient and outpatient clinical clerkship sites. No effect of clinical site assignment on NBME scores was demonstrated, suggesting that inter-site consistency occurred for the performance outcomes measured. Similar assessments can help other clerkship directors monitor clinical sites for quality of their educational programs, which is increasingly important as medical schools expand their clinical sites to community-based programs.

Assessment of Medicaid Managed Behavioral Health Care for Persons With Serious Mental Illness

This five-site study compared Medicaid managed behavioral health programs and fee-for-service programs on use and quality of services, satisfaction, and symptoms and functioning of adults with serious mental illness. Adults with serious mental illness in managed care programs (N=958) and fee-for-service programs (N=1,011) in five states were interviewed after the implementation of managed care and six months later. After a multiple regression to standardize the groups for case mix differences, a meta-analysis using a random-effects model was conducted, and bioequivalence methods were used to determine whether differences were significant for clinical or policy purposes. A significantly smaller proportion of the managed care group received inpatient care (5.7 percent compared with 11.5 percent). The managed care group received significantly more hours of primary care (4.9 compared with 4.5 hours) and was significantly less healthy. However, none of these differences exceed the bioequivalence criterion of 5 percent. Managed care and fee for service were "not different but not equivalent" on 20 of 34 dependent variables. Cochrane's Q statistic, which measured intersite consistency, was significant for 20 variables. Managed care and fee-for-service Medicaid programs did not differ on most measures; however, a lack of sufficient power was evident for many measures. Full endorsement of managed care for vulnerable populations will require further research that assumes low penetration rates and intersite variability.

Intersite Consistency as a Measurement of Programmatic Evaluation in a Medicine Clerkship with Multiple, Geographically Separated Sites

This study introduces "intersite consistency" as a measurement of programmatic evaluation and demonstrates its feasibility and construct validity. Student data in our multisite, geographically separated clerkship were collected prospectively over a ten-year period (1990-2000). We calculated mean scores for each clerkship measurement and analyzed these data on both a yearly and a 10-year cumulative basis. Analyses of variance (ANOVA) and linear regression were used for statistical analysis. Data for 1,632 (98%) students were included in our study. During this ten-year study period, we had 22 different on-site clerkship directors at seven clerkship sites. ANOVA and linear regression of year-to-year and cumulative data did not demonstrate an effect of site on student outcomes. Intersite consistency can be used as one measure of programmatic evaluation for multisite clerkships.

Impact of microbiology practice on cumulative prevalence of respiratory tract bacteria in patients with cystic fibrosis.

Investigators participating in the Epidemiologic Study of Cystic Fibrosis project began to collect microbiological, pulmonary, and nutritional data on cystic fibrosis (CF) patients at 180 North American sites in 1994. Part of this study was a survey undertaken in August 1995 to determine microbiology laboratory practices with regard to pulmonary specimens from CF patients. The survey included a section on test ordering, completed by a site clinician, and a section on test performance and reporting, completed by each site's clinical microbiology laboratory staff. Seventy-nine percent of the surveys were returned. There was intersite consistency of microbiology laboratory practices in most cases. The majority of sites follow most of the CF Foundation consensus conference recommendations. There were differences in the frequency at which specimens for culture were obtained, in the use of selective media for Staphylococcus aureus and Haemophilus influenzae, and in the use of a prolonged incubation for Burkholderia cepacia. These variations in practice contribute to prevalence differences among sites and may result in differences in clinical care.

Tectonic implications of paleomagnetic research into Upper Cretaceous magmatic rocks in the Apuseni Mountains, Romania

The paleomagnetism of Upper Cretaceous magmatic rocks from 47 collecting sites (172 samples, 692 specimens) in the Apuseni Mountains was studied. After AF cleaning, characteristic magnetizations were identified for various collecting areas in the study zone, which defined a few spatial and temporal units for which paleomagnetic poles could be derived statistically. At 21 sampling sites the paleomagnetic directions showed a high level of intrasite and intersite consistency, with a mean direction of I f = −38° and D f = −100°, with α 95 = 6°. The paleomagnetic results show that to reach their present-day position the Apuseni Mountains moved to the north, around 14° with respect to Europe, or around 25° with respect to the geographic poles, between the Campanian and, probably, Late Miocene, while a clockwise rotation, of around 80°, was taking place.

Paleomagnetic evidence for tectonic rotation of Jurassic plutons in Blue Mountains, eastern Oregon

To test the hypothesis that pre‐Tertiary rocks of eastern Oregon have undergone clockwise rotation, a paleomagnetic investigation was made of granitic and high‐temperature metamorphic rocks of the upper Jurassic‐Lower Cretaceous intrusive suite of the Blue Mountains province, northeastern Oregon. At nine sampling sites in three widely separated regions the paleomagnetic directions showed a high level of intrasite and intersite consistency, with a mean direction of I = 63°, D = 30°, a95 = 10°. The observed inclination is consistent with that expected on the basis of paleomagnetic results for stable North America, indicating that the intrusives have not undergone northward translation since cooling. The paleomagnetic declination, on the other hand, indicates that the region sampled has undergone a clockwise rotation of 60°±29° relative to the stable craton. Two of the bodies sampled are intrusive into the exotic Seven Devils terrane, and the remaining body is intrusive into the tectonic melange of a separate arc‐trench terrane, which may well have been the North American margin. The consistency in magnetic directions between the three regions (k = 151) indicates that suturing of the exotic terrane took place prior to the emplacement of the intrusives. The early Tertiary Clarno Formation situated to the west of the region sampled has rotated substantially less than the Blue Mountains [Beck et al., 1978], indicating that much of the rotation of the Blue Mountains occurred prior to the Eocene. To account for the observed rotations, we discuss two alternative models. In the first, rotation accompanies back arc rifting associated with the Early Cretaceous westward rifting of the Klamath Mountains. In the second, rotation reflects Late Cretaceous regional dextral shearing in the Blue Mountain area, which is located several hundred kilometers south of major right lateral faults known to extend from Alaska down at least to northern Washington.