Construct Responsiveness Research Articles

An Item Response Tree Model for Items with Multiple‐Choice and Constructed‐Response Parts

AbstractTraditional IRT and IRTree models are not appropriate for analyzing the item that simultaneously consists of multiple‐choice (MC) task and constructed‐response (CR) task in one item. To address this issue, this study proposed an item response tree model (called as IRTree‐MR) to accommodate items that contain different response types at different steps and multiple different cognitive processes behind each score to effectively investigate the cognitive process and achieve a more accurate evaluation of examinees. The proposed model employs appropriate processing function for each task and allows multiple paths to an observed outcome. The simulation studies were conducted to evaluate the performance of the proposed IRTree‐MR, and results show the proposed model outperforms the traditional IRT model in terms of parameters recovery and model‐fit. Moreover, an empirical study was carried out to verify the advantages of the proposed model.

FEW questions, many answers: using machine learning to assess how students connect food–energy–water (FEW) concepts

There is growing support and interest in postsecondary interdisciplinary environmental education, which integrates concepts and disciplines in addition to providing varied perspectives. There is a need to assess student learning in these programs as well as rigorous evaluation of educational practices, especially of complex synthesis concepts. This work tests a text classification machine learning model as a tool to assess student systems thinking capabilities using two questions anchored by the Food-Energy-Water (FEW) Nexus phenomena by answering two questions (1) Can machine learning models be used to identify instructor-determined important concepts in student responses? (2) What do college students know about the interconnections between food, energy, and water, and how have students assimilated systems thinking into their constructed responses about FEW? Reported here is a broad range of model performances across 26 text classification models associated with two different assessment items, with model accuracy ranging from 0.755 to 0.992. Expert-like responses were infrequent in our dataset compared to responses providing simpler, incomplete explanations of the systems presented in the question. For those students moving from describing individual effects to multiple effects, their reasoning about the mechanism behind the system indicates advanced systems thinking ability. Specifically, students exhibit higher expertise in explaining changing water usage than discussing trade-offs for such changing usage. This research represents one of the first attempts to assess the links between foundational, discipline-specific concepts and systems thinking ability. These text classification approaches to scoring student FEW Nexus Constructed Responses (CR) indicate how these approaches can be used, in addition to several future research priorities for interdisciplinary, practice-based education research. Development of further complex question items using machine learning would allow evaluation of the relationship between foundational concept understanding and integration of those concepts as well as a more nuanced understanding of student comprehension of complex interdisciplinary concepts.

Impact of Violating Unidimensionality on Rasch Calibration for Mixed-Format Tests

ABSTRACT Mixed-format tests that include both multiple-choice (MC) and constructed-response (CR) items have become widely used in many large-scale assessments. When an item response theory (IRT) model is used to score a mixed-format test, the unidimensionality assumption may be violated if the CR items measure a different construct from that measured by MC items. In this simulation study, we explore the impact of different degrees of violation on the unidimensionality assumption on Rasch item parameter recovery and model-data fit statistics. Our results suggest that, when the unidimensionality assumption is violated for mixed-format tests, both estimated Rasch item parameters and model-data fit statistics are distorted, with greater impact corresponding to higher degrees of violations. The findings from this study may provide guidance for test practitioners for working with mixed-format tests within the Rasch context.

Enhancing Rubric Development in Science Education through Topic Modeling Techniques

Content experts conduct traditional approaches to developing rubrics for constructed response (CR) items, which is a labor-intensive process. We aim to illustrate the potential benefits of utilizing topic modeling techniques to improve the efficiency of the rubric workload. The results of this study reveal that over half of the keywords in the rubric corresponded with the top 20 words extracted from the supervised latent Dirichlet allocation (sLDA) model. The rubric-based scores assigned by human raters showed medium-to-high associations with the sLDA-predicted scores. The sLDA approach offers supporting evidence for refining the rubric concerning criteria for word usage and score assignments. It accomplishes this by concurrently incorporating the predetermined rubric of content experts and the actual responses provided by students.

Analysis of Mixed-Format Assessments Using Measurement Models and Topic Modeling

ABSTRACT It is common to find mixed-format data results from the use of both multiple-choice (MC) and constructed-response (CR) questions on assessments. Dealing with these mixed response types involves understanding what the assessment is measuring, and the use of suitable measurement models to estimate latent abilities. Past research in educational measurement, however, has often overlooked the written responses in CR items after analyzing the response scores. This study presents a method for bridging this gap by using a topic model called latent Dirichlet allocation to uncover the structure in written answers and then use that information to augment results from traditional measurement models. In this study, a five-step framework is employed for assessing both the examinees’ latent abilities and the internal structure of the written responses obtained from mixed-format assessments. An empirical dataset obtained from Grade 8 examinees in a southeastern state on an English language arts assessment is used for illustration. Based on the dimensionality of the assessment, a unidimensional partial credit model and two multidimensional bi-factor models were fit to the data. A comparison of results from these analyses suggests that a constrained bi-factor model was most useful for detecting the mixed-format score patterns, and that a 5-topic model was determined for the textual responses. The topic distributions were found to be related to the latent abilities estimated from the constrained bi-factor model. This framework combines both the score patterns and textual responses, and highlights the utility of combining the traditional response analysis with a start-of-art language model.

Developing Student Expertise in Evolution: Cognitive Construals Complement Key Concepts in Student Representations.

Genetic variation is historically challenging for undergraduate students to master, potentially due to its grounding in both evolution and genetics. Traditionally, student expertise in genetic variation has been evaluated using Key Concepts. However, Cognitive Construals may add to a more nuanced picture of students' developing expertise. Here, we analyze the occurrence of Key Concepts and Cognitive Construals among three types of student representations: interviews, drawn models, and constructed responses (CRs). Our mixed-methods analysis indicates that differential survival and differential reproduction occur more often in interviews than in CRs. In our interviews, presence of Cognitive Construals indicate varying levels of understanding of genetic variation, but we were not able to detect Cognitive Construals in students' models or CRs. Finally, our analyses of both Key Concepts and Cognitive Construals in student representations indicate that Cognitive Construals can co-occur with any number of Key Concepts, and that the presence of Construal-based language alone does not seem to correlate to the expert nature of a response. Taken together, our results highlight the need for instructors to avoid treating Construal-based language as implicit disconnects in student understanding, and to use multiple methods to gain a holistic picture of student expertise.

Detecting Rater Bias in Mixed-Format Assessments

ABSTRACT Mixed-format assessments made up of multiple-choice (MC) items and constructed response (CR) items that are scored using rater judgments include unique psychometric considerations. When these item types are combined to estimate examinee achievement, information about the psychometric quality of each component can depend on that of the other. For example, the presence of differential item functioning (DIF) in MC items could compromise the sensitivity of rater effect indicators, such as differential rater functioning (DRF) indices. Likewise, the presence of DRF could compromise the sensitivity of DIF indices. We used real data and a simulation study to consider the impact of DIF on DRF indices, and the impact of DRF on DIF indices. Results indicate some interaction between DIF and DRF that varies across data collection designs. We consider the implications of these results for research and practice.

Construct Validity of the Constructed Response Test Tasks in School-Based Assessment

This study aimed to examine the constructed response (CR) test tasks in school-based assessment with a focus on construct validity. To this end, the researcher collected 771 CR test items from the midterm and final exam papers of the 2021 academic year from 37 high schools across the country. Three Korean raters participated in analyzing the characteristics of the test tasks based on the modified Bachman and Palmer’s test method facet, and several corpus analysis tools including WordSmith 8.0, Range, LCA, L2SCA and readability calculators were applied. The result of the analysis clearly demonstrates a gap between the practice and the published guidelines of introducing CR test tasks. The analysis of the task characteristics showed that the CR test is mainly intended to assess students’ reading skills rather than writing skills, largely through short answer or completion form of the CR test tasks. The corpus analysis of the language input showed that lexical difficulty, diversity, and complexity, syntactic complexity, and textual difficulty varied depending on three source types: the textbook, the NCSAT (mock CSAT) and others. The findings of the study can give several pedagogical implications in regard to the CR test tasks in school-based assessments.

Exploring examinees' responses to constructed response items with a supervised topic model.

Textual data are increasingly common in test data as many assessments include constructed response (CR) items as indicators of participants' understanding. The development of techniques based on natural language processing has made it possible for researchers to rapidly analyse large sets of textual data. One family of statistical techniques for this purpose are probabilistic topic models. Topic modelling is a technique for detecting the latent topic structure in a collection of documents and has been widely used to analyse texts in a variety of areas. The detected topics can reveal primary themes in the documents, and the relative use of topics can be useful in investigating the variability of the documents. Supervised latent Dirichlet allocation (SLDA) is a popular topic model in that family that jointly models textual data and paired responses such as could occur with participants' textual answers to CR items and their rubric-based scores. SLDA has an assumption of a homogeneous relationship between textual data and paired responses across all documents. This approach, while useful for some purposes, may not be satisfied for situations in which a population has subgroups that have different relationships. In this study, we introduce a new supervised topic model that incorporates finite-mixture modelling into the SLDA. This new model can detect latent groups of participants that have different relationships between their textual responses and associated scores. The model is illustrated with an example from an analysis of a set of textual responses and paired scores from a middle grades assessment of science inquiry knowledge. A simulation study is presented to investigate the performance of the proposed model under practical testing conditions.

Sequential Bayesian Ability Estimation Applied to Mixed-Format Item Tests.

Large-scale tests often contain mixed-format items, such as when multiple-choice (MC) items and constructed-response (CR) items are both contained in the same test. Although previous research has analyzed both types of items simultaneously, this may not always provide the best estimate of ability. In this paper, a two-step sequential Bayesian (SB) analytic method under the concept of empirical Bayes is explored for mixed item response models. This method integrates ability estimates from different item formats. Unlike the empirical Bayes method, the SB method estimates examinees' posterior ability parameters with individual-level sample-dependent prior distributions estimated from the MC items. Simulations were used to evaluate the accuracy of recovery of ability and item parameters over four factors: the type of the ability distribution, sample size, test length (number of items for each item type), and person/item parameter estimation method. The SB method was compared with a traditional concurrent Bayesian (CB) calibration method, EAPsum, that uses scaled scores for summed scores to estimate parameters from the MC and CR items simultaneously in one estimation step. From the simulation results, the SB method showed more accurate and reliable ability estimation than the CB method, especially when the sample size was small (150 and 500). Both methods presented similar recovery results for MC item parameters, but the CB method yielded a bit better recovery of the CR item parameters. The empirical example suggested that posterior ability estimated by the proposed SB method had higher reliability than the CB method.

Covariational reasoning and item context affect language in undergraduate mass balance written explanations.

Mass balance (MB) reasoning offers a rich topic for examination of students' scientific thinking and skills, as it requires students to account for multiple inputs and outputs within a system and apply covariational reasoning. Using previously validated constructed response prompts for MB, we examined 1,920 student-constructed responses (CRs) aligned to an emerging learning progression to determine how student language changes from low (1) to high (4) covariational reasoning levels. As students' abilities and thinking change with Context, we used the same general prompt in six physiological contexts. We asked how Level and Context affect student language and what language is conserved across Contexts at higher reasoning Levels. Using diversity methods, we found student language becomes more similar as covariational reasoning level increases. Using text analysis, we found context-dependent words at each Level; however, the type of context words changed. Specifically, at Level 1, students used context words that are tangential to MB reasoning, while Level 4 responses used words that specify inputs and outputs for the given Item Context. Further, at Level 4, students shared 30% of language across the six contexts and leveraged context-independent words including rate, equal, and some form of slower/lower/smaller. Together, these data demonstrate that Context affects undergraduate MB language at all covariational reasoning levels, but that the language becomes more specific and similar as Level increases. These findings encourage instructors to foster context-independent, comparative, and summative language during instruction to functionally build MB and covariational reasoning skills across contexts.NEW & NOTEWORTHY This article builds on the work of Scott et al. (Scott EE, Cerchiara J, McFarland JL, Wenderoth MP, Doherty JH. J Res Sci Teach 1: 37, 2023) and Shiroda et al. (Shiroda M, Fleming MP, Haudek KC. Front Educ 8: 989836, 2023) to quantitatively examine student language in written explanations of mass balance across six contexts using constructed response assessments. These results present an evaluation of student mass balance language and provide researchers and practitioners with tools to assist students in constructing scientific mass balance reasoning explanations.

혼합형 가교문항 구성 방법이 혼합형 검사 동등화에 미치는 영향

With the introduction of innovative items in the computer-based NAEA, the item format became diversified, and the current method of selecting anchor items from multiple-choice (MC) items became burdensome for test developers. The purpose of this study is to examine whether a mixed-format anchor items including both MC and constructed-response (CR) items could hold the stability of equating in the NAEA, and give desirable direction to expand the range of anchor items and to make test construct flexible. A simulation study was conducted based on a common-item non-equivalent group design with the test conditions of the Korean subject in the NAEA. The treatment conditions include (1) a type of anchor item set (a single MC or a mixed), (2) a composition type of mixed anchor item (MC:CR = 1:1, 3:2, or 3:4), (3) a length of mixed anchor items within a test (25%, 22%, or 19%), and (4) the presence of multidimensionality according to the item format were assumed. The main result is that there was no significant difference in the equating error between the single MC and the mixed anchor item condition. Specifically the equating error of the single MC was smaller than that of the mixed-format cases where the ratios between MC and CR were 1:1 and 3:2. In the 3:4-ratio mixed condition, the equating error was smaller than that of the single MC condition. A length of mixed anchor items within a test showed somewhat mixed results, probably because there was not much variation between the conditions in the composition ratio, about 20% in test. When there was multidimensionality according to the item format in the test, the equating error increased significantly with mixed anchor items compared to that of unidimensional test condition. In sum, about 20% of the mixed anchor items with slightly higher CR ratio was found to be suitable in the unidimensional test and caution was called for in using mixed anchor items in multidimensional test.

Employer Responses to Poaching on Employee Productivity: The Mediating Role of Organizational Agility in Technology Companies

This study investigates the impact of an organization’s response to poaching on employee productivity while considering the mediating role of organizational agility, contributing to the sustainability literature. The study aimed to explore the impact of three responses toward poaching, including the constructive response (CR), defensive response (DR), and retaliatory response (RR), on employee productivity. The survey-based approach collected data from companies in the Pakistani service sector, an essential step toward understanding sustainable human resource practices in developing countries. Regression analysis was used to analyze the collected data, revealing significant relationships between predictor variables and employee productivity. Specifically, Model 1 found a significant negative relationship between CR and productivity, whereas DR showed a positive and significant association with productivity. The study also considered the unique characteristics of the Pakistani service sector to explain the observed negative and positive relationships between CR and DR, respectively, and employee productivity. Model 2 improved the model fit by introducing organizational agility as a predictor variable, demonstrating a positive and significant relationship between productivity and all four predictor variables. The study suggests that the positive relationship between organizational agility and productivity could be due to agile organizations being more adept at responding to changing market conditions and demands, resulting in improved overall performance. This study provides valuable insights into the impact of an organization’s response to poaching on employee productivity while considering the mediating role of organizational agility. It contributes to the sustainability literature.

Ecological diversity methods improve quantitative examination of student language in short constructed responses in STEM

We novelly applied established ecology methods to quantify and compare language diversity within a corpus of short written student texts. Constructed responses (CRs) are a common form of assessment but are difficult to evaluate using traditional methods of lexical diversity due to text length restrictions. Herein, we examined the utility of ecological diversity measures and ordination techniques to quantify differences in short texts by applying these methods in parallel to traditional text analysis methods to a corpus of previously studied college student CRs. The CRs were collected at two time points (Timing), from three types of higher-ed institutions (Type), and across three levels of student understanding (Thinking). Using previous work, we were able to predict that we would observe the most difference based on Thinking, then Timing and did not expect differences based on Type allowing us to test the utility of these methods for categorical examination of the corpus. We found that the ecological diversity metrics that compare CRs to each other (Whittaker’s beta, species turnover, and Bray–Curtis Dissimilarity) were informative and correlated well with our predicted differences among categories and other text analysis methods. Other ecological measures, including Shannon’s and Simpson’s diversity, measure the diversity of language within a single CR. Additionally, ordination provided meaningful visual representations of the corpus by reducing complex word frequency matrices to two-dimensional graphs. Using the ordination graphs, we were able to observe patterns in the CR corpus that further supported our predictions for the data set. This work establishes novel approaches to measuring language diversity within short texts that can be used to examine differences in student language and possible associations with categorical data.

Validation of automated scoring for learning progression-aligned Next Generation Science Standards performance assessments

IntroductionThe Framework for K-12 Science Education promotes supporting the development of knowledge application skills along previously validated learning progressions (LPs). Effective assessment of knowledge application requires LP-aligned constructed-response (CR) assessments. But these assessments are time-consuming and expensive to score and provide feedback for. As part of artificial intelligence, machine learning (ML) presents an invaluable tool for conducting validation studies and providing immediate feedback. To fully evaluate the validity of machine-based scores, it is important to investigate human-machine score consistency beyond observed scores. Importantly, no formal studies have explored the nature of disagreements between human and machine-assigned scores as related to LP levels.MethodsWe used quantitative and qualitative approaches to investigate the nature of disagreements among human and scores generated by two approaches to machine learning using a previously validated assessment instrument aligned to LP for scientific argumentation.ResultsWe applied quantitative approaches, including agreement measures, confirmatory factor analysis, and generalizability studies, to identify items that represent threats to validity for different machine scoring approaches. This analysis allowed us to determine specific elements of argumentation practice at each level of the LP that are associated with a higher percentage of misscores by each of the scoring approaches. We further used qualitative analysis of the items identified by quantitative methods to examine the consistency between the misscores, the scoring rubrics, and student responses. We found that rubrics that require interpretation by human coders and items which target more sophisticated argumentation practice present the greatest threats to the validity of machine scores.DiscussionWe use this information to construct a fine-grained validity argument for machine scores, which is an important piece because it provides insights for improving the design of LP-aligned assessments and artificial intelligence-enabled scoring of those assessments.

Rubric development for AI-enabled scoring of three-dimensional constructed-response assessment aligned to NGSS learning progression

IntroductionThe Framework for K-12 Science Education (the Framework) and the Next- Generation Science Standards (NGSS) define three dimensions of science: disciplinary core ideas, scientific and engineering practices, and crosscutting concepts and emphasize the integration of the three dimensions (3D) to reflect deep science understanding. The Framework also emphasizes the importance of using learning progressions (LPs) as roadmaps to guide assessment development. These assessments capable of measuring the integration of NGSS dimensions should probe the ability to explain phenomena and solve problems. This calls for the development of constructed response (CR) or open-ended assessments despite being expensive to score. Artificial intelligence (AI) technology such as machine learning (ML)-based approaches have been utilized to score and provide feedback on open-ended NGSS assessments aligned to LPs. ML approaches can use classifications resulting from holistic and analytic coding schemes for scoring short CR assessments. Analytic rubrics have been shown to be easier to evaluate for the validity of ML-based scores with respect to LP levels. However, a possible drawback of using analytic rubrics for NGSS-aligned CR assessments is the potential for oversimplification of integrated ideas. Here we describe how to deconstruct a 3D holistic rubric for CR assessments probing the levels of an NGSS-aligned LP for high school physical sciences.MethodsWe deconstruct this rubric into seven analytic categories to preserve the 3D nature of the rubric and its result scores and provide subsequent combinations of categories to LP levels.ResultsThe resulting analytic rubric had excellent human- human inter-rater reliability across seven categories (Cohen’s kappa range 0.82–0.97). We found overall scores of responses using the combination of analytic rubric very closely agreed with scores assigned using a holistic rubric (99% agreement), suggesting the 3D natures of the rubric and scores were maintained. We found differing levels of agreement between ML models using analytic rubric scores and human-assigned scores. ML models for categories with a low number of positive cases displayed the lowest level of agreement.DiscussionWe discuss these differences in bin performance and discuss the implications and further applications for this rubric deconstruction approach.

Automated Writing Assessments Measure Undergraduate Learning after Completion of a Computer-Based Cellular Respiration Tutorial.

The focus of biology education has shifted from memorization to conceptual understanding of core biological concepts such as matter and energy relationships. To examine undergraduate learning about matter and energy, we incorporated constructed-response (CR) questions into an interactive computer-based tutorial. The objective of this tutorial is to teach students about matter and energy and help dispel common misconceptions through the context of cellular respiration. We used a constructed-response classifier (CRC) tool to categorize ideas in responses to three CR questions and measure changes in student thinking about cellular respiration. Our data set includes 841 undergraduates from 19 geographically diverse institutions including two-year colleges, primarily undergraduate institutions, and research-intensive colleges and universities. We found students from all institution types included more scientific ideas in CRs post-tutorial. Students used an average of 2.1 ideas in CRs and frequently used both scientific and developing ideas. We found this mixed thinking persisted after the tutorial regardless of institution type. Students’ multiple-choice (MC) selections were correlated with their CRs, but CRs revealed more mixed thinking than would be inferred from MC responses. Our study shows a CRC tool can measure student learning after a computer-based tutorial and provides more complete information than MC responses.

Beyond Agreement: Exploring Rater Effects in Large-Scale Mixed Format Assessments

ABSTRACT Scoring procedures for the constructed-response (CR) items in large-scale mixed-format educational assessments often involve checks for rater agreement or rater reliability. Although these analyses are important, researchers have documented rater effects that persist despite rater training and that are not always detected in rater agreement and reliability analyses, such as severity/leniency, centrality/extremism, and biases. Left undetected, these effects pose threats to fairness. We illustrate how rater effects analyses can be incorporated into scoring procedures for large-scale mixed-format assessments. We used data from the National Assessment of Educational Progress (NAEP) to illustrate relatively simple analyses that can provide insight into patterns of rater judgment that may warrant additional attention. Our results suggested that the NAEP raters exhibited generally defensible psychometric properties, while also exhibiting some idiosyncrasies that could inform scoring procedures. Similar procedures could be used operationally to inform the interpretation and use of rater judgments in large-scale mixed-format assessments.

Examining the Impacts of Ignoring Rater Effects in Mixed‐Format Tests

AbstractThe use of mixed‐format tests made up of multiple‐choice (MC) items and constructed response (CR) items is popular in large‐scale testing programs, including the National Assessment of Educational Progress (NAEP) and many district‐ and state‐level assessments in the United States. Rater effects, or raters’ scoring tendencies that result in performances receiving different scores than are warranted given their quality, are concerns for the interpretation of scores on CR items. However, there are few published studies in which researchers have systematically considered the impact of ignoring rater effects when they are present on estimates of student ability using large‐scale mixed‐format assessments. Using results from an analysis of NAEP data, we systematically explored the impacts of rater effects on student achievement estimates. Our results suggest that in conditions that reflect many large‐scale mixed‐format assessments, directly modeling rater effects yields more accurate student achievement estimates than estimation procedures that do not incorporate raters. We consider the implications of our findings for research and practice.

Integrating a Statistical Topic Model and a Diagnostic Classification Model for Analyzing Items in a Mixed Format Assessment.

Selected response items and constructed response (CR) items are often found in the same test. Conventional psychometric models for these two types of items typically focus on using the scores for correctness of the responses. Recent research suggests, however, that more information may be available from the CR items than just scores for correctness. In this study, we describe an approach in which a statistical topic model along with a diagnostic classification model (DCM) was applied to a mixed item format formative test of English and Language Arts. The DCM was used to estimate students’ mastery status of reading skills. These mastery statuses were then included in a topic model as covariates to predict students’ use of each of the latent topics in their written answers to a CR item. This approach enabled investigation of the effects of mastery status of reading skills on writing patterns. Results indicated that one of the skills, Integration of Knowledge and Ideas, helped detect and explain students’ writing patterns with respect to students’ use of individual topics.