Interdisciplinary Science Assessment of Carbon Cycling: Construct Validity Evidence Based on Internal Structure

Abstract

Growing interest in interdisciplinary (ID) understanding has led to the recent development of four ID assessments, none of which have previously been comprehensively validated. Sources of evidence for the validity of tests include construct validity, such as the internal structure of the test. ID tests may (and should) test both disciplinary (D) and ID understanding, and the internal structure can be examined to determine whether the theoretical relationship among D and ID knowledge is supported by empirical evidence. In this paper, we present an analysis of internal structure of ISACC, an ID assessment of carbon cycling, selected because the developers included both disciplinary and ID items and posited a relationship among these two types of knowledge. Responses from 454 high school and college students were analyzed using confirmatory factor analyses (CFA) with the Mplus software for internal structure focusing on dimensionality, as well as functioning by gender and race/ethnicity, and reliability. CFA confirmed that the underlying structure of the ISACC best matches a two-factor path model, supporting the developers’ theoretical hypothesis of the impact of disciplinary understanding on interdisciplinary understanding. No gender effect was found in the factor structure of the ISACC, indicating that females and males have similar performance. Race/ethnicity performance was similar to other science assessments, revealing possible ethnicity bias on the part of the instrument. The reliability coefficients for the two-factor path model were found to be sufficient. This study highlights the importance of the internal structure of test instruments as a source of validity evidence and models a procedure to assess internal structure.