Constructed-response Items Research Articles

Learning to Score: A Coding System for Constructed Response Items via Interactive Clustering

Constructed response items that require the student to give more detailed and elaborate responses are widely applied in large-scale assessments. However, the hand-craft scoring with a rubric for massive responses is labor-intensive and impractical due to rater subjectivity and answer variability. The automatic response coding method, such as the automatic scoring of short answers, has become a critical component of the learning and assessment system. In this paper, we propose an interactive coding system called ASSIST to efficiently score student responses with expert knowledge and then generate an automatic score classifier. First, the ungraded responses are clustered to generate specific codes, representative responses, and indicator words. The constraint set based on feedback from experts is taken as training data in metric learning to compensate for machine bias. Meanwhile, the classifier from responses to code is trained according to the clustering results. Second, the experts review each coded cluster with the representative responses and indicator words to score a rating. The coded cluster and score pairs will be validated to ensure inter-rater reliability. Finally, the classifier is available for scoring a new response with out-of-distribution detection, which is based on the similarity between response representation and class proxy, i.e., the weight of class in the last linear layer of the classifier. The originality of the system developed stems from the interactive response clustering procedure, which involves expert feedback and an adaptive automatic classifier that can identify new response classes. The proposed system is evaluated on our real-world assessment dataset. The results of the experiments demonstrate the effectiveness of the proposed system in saving human effort and improving scoring performance. The average improvements in clustering quality and scoring accuracy are 14.48% and 18.94%, respectively. Additionally, we reported the inter-rater reliability, out-of-distribution rate, and cluster statistics, before and after interaction.

Automated Scoring of Constructed Response Items in Math Assessment Using Large Language Models

Automated Scoring of Constructed Response Items in Math Assessment Using Large Language Models

Impact of Violating Unidimensionality on Rasch Calibration for Mixed-Format Tests

ABSTRACT Mixed-format tests that include both multiple-choice (MC) and constructed-response (CR) items have become widely used in many large-scale assessments. When an item response theory (IRT) model is used to score a mixed-format test, the unidimensionality assumption may be violated if the CR items measure a different construct from that measured by MC items. In this simulation study, we explore the impact of different degrees of violation on the unidimensionality assumption on Rasch item parameter recovery and model-data fit statistics. Our results suggest that, when the unidimensionality assumption is violated for mixed-format tests, both estimated Rasch item parameters and model-data fit statistics are distorted, with greater impact corresponding to higher degrees of violations. The findings from this study may provide guidance for test practitioners for working with mixed-format tests within the Rasch context.

New Tests of Rater Drift in Trend Scoring

ABSTRACT Trend scoring constructed response items (i.e. rescoring Time A responses at Time B) gives rise to two-way data that follow a product multinomial distribution rather than the multinomial distribution that is usually assumed. Recent work has shown that the difference in sampling model can have profound negative effects on statistics usually used to monitor rater drift. In this paper, three statistics, termed E-statistics, that account for the product-multinomial sampling by comparing conditional distributions, are introduced. A simulation compares performance with the paired t-test and Stuart’s Q in detecting rater drift. Both the paired t-test and Q suffered extreme Type I error inflation for certain rescore study designs. The new E-statistics maintained good Type I error control and had good power to detect rater drift across occasions.

Resolving and Re-Scoring Constructed Response Items in Mixed-Format Assessments: An Exploration of Three Approaches

ABSTRACT We explored three approaches to resolving or re-scoring constructed-response items in mixed-format assessments: rater agreement, person fit, and targeted double scoring (TDS). We used a simulation study to consider how the three approaches impact the psychometric properties of student achievement estimates, with an emphasis on person fit. We found that although TDS is not specifically designed to identify discrepant or misfitting score profiles, this approach can improve person fit for students who are also within a critical range of achievement. Our findings also suggest that TDS alone is not sufficient to provide a comprehensive picture of measurement quality because it does not directly consider person fit. We suggest that researchers and practitioners consider combining TDS with other approaches to score resolution. We consider the implications of our findings for research and practice.

Combining machine translation and automated scoring in international large-scale assessments

BackgroundArtificial intelligence (AI) is rapidly changing communication and technology-driven content creation and is also being used more frequently in education. Despite these advancements, AI-powered automated scoring in international large-scale assessments (ILSAs) remains largely unexplored due to the scoring challenges associated with processing large amounts of multilingual responses. However, due to their low-stakes nature, ILSAs are an ideal ground for innovations and exploring new methodologies.MethodsThis study proposes combining state-of-the-art machine translations (i.e., Google Translate & ChatGPT) and artificial neural networks (ANNs) to mitigate two key concerns of human scoring: inconsistency and high expense. We applied AI-based automated scoring to multilingual student responses from eight countries and six different languages, using six constructed response items from TIMSS 2019.ResultsAutomated scoring displayed comparable performance to human scoring, especially when the ANNs were trained and tested on ChatGPT-translated responses. Furthermore, psychometric characteristics derived from machine scores generally exhibited similarity to those obtained from human scores. These results can be considered as supportive evidence for the validity of automated scoring for survey assessments.ConclusionsThis study highlights that automated scoring integrated with the recent machine translation holds great promise for consistent and resource-efficient scoring in ILSAs.

Analysis of Mixed-Format Assessments Using Measurement Models and Topic Modeling

ABSTRACT It is common to find mixed-format data results from the use of both multiple-choice (MC) and constructed-response (CR) questions on assessments. Dealing with these mixed response types involves understanding what the assessment is measuring, and the use of suitable measurement models to estimate latent abilities. Past research in educational measurement, however, has often overlooked the written responses in CR items after analyzing the response scores. This study presents a method for bridging this gap by using a topic model called latent Dirichlet allocation to uncover the structure in written answers and then use that information to augment results from traditional measurement models. In this study, a five-step framework is employed for assessing both the examinees’ latent abilities and the internal structure of the written responses obtained from mixed-format assessments. An empirical dataset obtained from Grade 8 examinees in a southeastern state on an English language arts assessment is used for illustration. Based on the dimensionality of the assessment, a unidimensional partial credit model and two multidimensional bi-factor models were fit to the data. A comparison of results from these analyses suggests that a constrained bi-factor model was most useful for detecting the mixed-format score patterns, and that a 5-topic model was determined for the textual responses. The topic distributions were found to be related to the latent abilities estimated from the constrained bi-factor model. This framework combines both the score patterns and textual responses, and highlights the utility of combining the traditional response analysis with a start-of-art language model.

Transformation Model of History Learning in Increasing Student Competency

Since Islamic boarding school-based madrasas maintain books as a source of history learning, teachers face serious challenges, form students' critical and chronological thinking skills through book study. This encourages a transformation of history learning, which has been taking place traditionally. This study aims to explore and describe book-based history teaching and learning to improve student competence. This research is qualitative, using a case study type. Data collection techniques use interviews, observation and documentation. Data is analyzed using relevant theories or previous research findings. Data validity is carried out by extending observations. This study found that book-based history learning was transformed by formulating personal-social goals without ignoring transformative goals, emphasizing a chronological approach in education, and developing CRI (Constructed Response Items) model evaluation questions. Book-based learning activities are different from using modern textbooks. Teachers' creative and innovative efforts are needed to realise learning objectives in madrasas through book learning. Apart from mastering the content, the national curriculum is also directed at developing critical, creative, communicative, collaborative and literacy skills.

랜덤집단 설계 IRT 검사 동등화의 표준오차 추정을 위한 델타 방법의 기능 진단

In the academic achievement and ability testing programs, item response theory (IRT) methods have been widely used for test equating. The standard error (SE) of equating indicates the amount of random error that is due to the sampling of examinees in estimating the population equivalent scores relationship. The purpose of the present study was to use computer simulations to examine the accuracy and performance of the delta method formulas for estimating SEs of IRT true score and observed score equating that have been presented under the random groups design. Test type (3PL model test, 3PL+GPC model test, and 3PL+GR model test) and sample size were considered as simulation factors. Main results were as follows. First, for all the test types, the theoretical SEs of IRT equating estimated by the delta method were very close to the empirical SEs computed using the simulated equated scores. Second, the SEs of IRT equating were reduced as the sample size increased, and they were, approximately, inversely proportional to the square root of the sample size. Third, except for the extreme (lowest or highest) test scores, the SEs of IRT observed score equating tended to be slightly smaller than the SEs of IRT true score equating. Fourth, on the average, the SEs of IRT equating for the 3PL model test, which consisted of 30 multiple-choice items, were smaller than those for the mixed-format tests, which consisted of the same 30 multiple-choice items and 10 constructed-response items.

A Comparison of Latent Semantic Analysis and Latent Dirichlet Allocation in Educational Measurement

Topic models are mathematical and statistical models used to analyze textual data. The objective of topic models is to gain information about the latent semantic space of a set of related textual data. The semantic space of a set of textual data contains the relationship between documents and words and how they are used. Topic models are becoming more common in educational measurement research as a method for analyzing students’ responses to constructed-response items. Two popular topic models are latent semantic analysis (LSA) and latent Dirichlet allocation (LDA). LSA uses linear algebra techniques, whereas LDA uses an assumed statistical model and generative process. In educational measurement, LSA is often used in algorithmic scoring of essays due to its high reliability and agreement with human raters. LDA is often used as a supplemental analysis to gain additional information about students, such as their thinking and reasoning. This article reviews and compares the LSA and LDA topic models. This article also introduces a methodology for comparing the semantic spaces obtained by the two models and uses a simulation study to investigate their similarities.

Exploring examinees' responses to constructed response items with a supervised topic model.

Textual data are increasingly common in test data as many assessments include constructed response (CR) items as indicators of participants' understanding. The development of techniques based on natural language processing has made it possible for researchers to rapidly analyse large sets of textual data. One family of statistical techniques for this purpose are probabilistic topic models. Topic modelling is a technique for detecting the latent topic structure in a collection of documents and has been widely used to analyse texts in a variety of areas. The detected topics can reveal primary themes in the documents, and the relative use of topics can be useful in investigating the variability of the documents. Supervised latent Dirichlet allocation (SLDA) is a popular topic model in that family that jointly models textual data and paired responses such as could occur with participants' textual answers to CR items and their rubric-based scores. SLDA has an assumption of a homogeneous relationship between textual data and paired responses across all documents. This approach, while useful for some purposes, may not be satisfied for situations in which a population has subgroups that have different relationships. In this study, we introduce a new supervised topic model that incorporates finite-mixture modelling into the SLDA. This new model can detect latent groups of participants that have different relationships between their textual responses and associated scores. The model is illustrated with an example from an analysis of a set of textual responses and paired scores from a middle grades assessment of science inquiry knowledge. A simulation study is presented to investigate the performance of the proposed model under practical testing conditions.

Historically restricted or historically empowered? Differences in access to historical content knowledge between low‐ and high‐SES pupils

AbstractIn the study presented in this article, the aim is to further the understanding regarding the differences between pupils (aged 15–16) from schools with low or high socio‐economic status (SES), regarding the amount and diversity of content knowledge in history that they have acquired by the end of compulsory schooling. Following a definition of historical content knowledge, we situate the concept in relation to other aspects of the history school subject. This is done to visualize historical content knowledge's central role in more complex aspects of the subject. The empirical material used in the study is pupils' responses on both selected and constructed response items on the Swedish national test in history. In the study, a combination of qualitative and quantitative approaches is used. The results show not only that pupils in low‐SES schools provide fewer examples of historical content knowledge. We can also establish that the historical content knowledge of pupils from high‐SES schools represents several perspectives while there are few perspectives present in the responses from pupils in low‐SES schools. The results are used to discuss how the differences between pupils in low‐ and high‐SES schools may affect their possibilities for educational success and active participation in society.

Sequential Bayesian Ability Estimation Applied to Mixed-Format Item Tests.

Large-scale tests often contain mixed-format items, such as when multiple-choice (MC) items and constructed-response (CR) items are both contained in the same test. Although previous research has analyzed both types of items simultaneously, this may not always provide the best estimate of ability. In this paper, a two-step sequential Bayesian (SB) analytic method under the concept of empirical Bayes is explored for mixed item response models. This method integrates ability estimates from different item formats. Unlike the empirical Bayes method, the SB method estimates examinees' posterior ability parameters with individual-level sample-dependent prior distributions estimated from the MC items. Simulations were used to evaluate the accuracy of recovery of ability and item parameters over four factors: the type of the ability distribution, sample size, test length (number of items for each item type), and person/item parameter estimation method. The SB method was compared with a traditional concurrent Bayesian (CB) calibration method, EAPsum, that uses scaled scores for summed scores to estimate parameters from the MC and CR items simultaneously in one estimation step. From the simulation results, the SB method showed more accurate and reliable ability estimation than the CB method, especially when the sample size was small (150 and 500). Both methods presented similar recovery results for MC item parameters, but the CB method yielded a bit better recovery of the CR item parameters. The empirical example suggested that posterior ability estimated by the proposed SB method had higher reliability than the CB method.

Causes of the Shortage of Physics Teachers in Croatia

Knowing the causes of the shortage of physics teachers in primary and secondary schools is necessary for the development of effective educational policies because the shortage of physics teachers is a global and persistent problem with negative consequences for the quality of education, but also for the survival of the physics profession as a whole. The aim of this study was to investigate, for the first time, the opinions of Croatian physics teachers on the causes of the deficit in their profession. For this purpose, we conducted a descriptive cross-sectional study using an online survey with Likert-type items and a constructed response item. A total of 390 respondents from all over Croatia participated in the survey, which is 29% of the total population of Croatian physics teachers in the 2022/2023 school year. According to their opinion, the causes of the shortage of Croatian physics teachers are related to the following: lack of incentives and support from the relevant institutions and bodies, the inadequacy of physics content in the curricula, the lack of motivation and negative attitude of students towards physics, impeded professional development, unequal opportunities, and challenges such as excessive workload. The results obtained provide a concrete basis for the development of an effective policy to solve the problem of the physics teacher shortage in Croatia and beyond by governments, universities, and schools.

English Learners and Constructed-Response Science Test Items Challenges and Opportunities

ABSTRACT This article reports on two quantitative studies of English learners’ (ELs) interactions with constructed-response items from a Grade 5 state science test. Study 1 investigated the relationships between the constructed-response item-level variables of English Reading Demand, English Writing Demand, and Background Knowledge Demand and the performance of ELs vs. non-ELs on those items. English Writing Demand was the strongest predictor of Differential Item Functioning favoring non-ELs over ELs for constructed-response items. In Study 2, we investigated the student-level variable of English language proficiency level and found that lower English language proficiency was related to greatly increased odds of omitting a response to a constructed-response item, even when controlling for science proficiency. These findings challenge the validity of scores on constructed-response test items as measures of ELs’ science proficiency.

Regularized Mislevy-Wu Model for Handling Nonignorable Missing Item Responses

Missing item responses are frequently found in educational large-scale assessment studies. In this article, the Mislevy-Wu item response model is applied for handling nonignorable missing item responses. This model allows that the missingness of an item depends on the item itself and a further latent variable. However, with low to moderate amounts of missing item responses, model parameters for the missingness mechanism are difficult to estimate. Hence, regularized estimation using a fused ridge penalty is applied to the Mislevy-Wu model to stabilize estimation. The fused ridge penalty function is separately defined for multiple-choice and constructed response items because previous research indicated that the missingness mechanisms strongly differed for the two item types. In a simulation study, it turned out that regularized estimation improves the stability of item parameter estimation. The method is also illustrated using international data from the progress in international reading literacy study (PIRLS) 2011 data.

혼합형 가교문항 구성 방법이 혼합형 검사 동등화에 미치는 영향

With the introduction of innovative items in the computer-based NAEA, the item format became diversified, and the current method of selecting anchor items from multiple-choice (MC) items became burdensome for test developers. The purpose of this study is to examine whether a mixed-format anchor items including both MC and constructed-response (CR) items could hold the stability of equating in the NAEA, and give desirable direction to expand the range of anchor items and to make test construct flexible. A simulation study was conducted based on a common-item non-equivalent group design with the test conditions of the Korean subject in the NAEA. The treatment conditions include (1) a type of anchor item set (a single MC or a mixed), (2) a composition type of mixed anchor item (MC:CR = 1:1, 3:2, or 3:4), (3) a length of mixed anchor items within a test (25%, 22%, or 19%), and (4) the presence of multidimensionality according to the item format were assumed. The main result is that there was no significant difference in the equating error between the single MC and the mixed anchor item condition. Specifically the equating error of the single MC was smaller than that of the mixed-format cases where the ratios between MC and CR were 1:1 and 3:2. In the 3:4-ratio mixed condition, the equating error was smaller than that of the single MC condition. A length of mixed anchor items within a test showed somewhat mixed results, probably because there was not much variation between the conditions in the composition ratio, about 20% in test. When there was multidimensionality according to the item format in the test, the equating error increased significantly with mixed anchor items compared to that of unidimensional test condition. In sum, about 20% of the mixed anchor items with slightly higher CR ratio was found to be suitable in the unidimensional test and caution was called for in using mixed anchor items in multidimensional test.

Diagnostic Tree Model을 활용한 수학 서술형 문항 인지진단 평가 적용 연구

This study aimed to analyze mathematics assessments with constructed-response items using diagnostic tree model (DTM), which accommodates polytomous responses with multiple strategies and obtain diagnostic information about examinees’ attribute mastery. A cognitive diagnostic assessment evaluating the “absolute inequality” domain with constructed-response items was designed based on the DTM and administered to students attending a science high school for gifted children. The findings from cognitive diagnosis of examinees indicate that students face more challenges in mastering the Cauchy-Schwarz inequality compared with the arithmetic-geometric mean inequality, with the greatest difficulty being in the application of the substitution method. The analysis revealed that for certain test items, the estimated guess and slip parameters vary depending on the nodes of the path selected by the chosen strategy, thus indicating that the difficulty to solve the problem may depend on the strategy that the students choose. Additionally, this study suggests several methods to visualize the item response of each examinee using a tree structure and estimated parameters for the DTM and to utilize it for individualized diagnostic feedback.

Examining how using dichotomous and partial credit scoring models influence sixth‐grade mathematical problem‐solving assessment outcomes

AbstractDetermining the most appropriate method of scoring an assessment is based on multiple factors, including the intended use of results, the assessment's purpose, and time constraints. Both the dichotomous and partial credit models have their advantages, yet direct comparisons of assessment outcomes from each method are not typical with constructed response items. The present study compared the impact of both scoring methods on the internal structure and consequential validity of a middle‐grades problem‐solving assessment called the problem solving measure for grade six (PSM6). After being scored both ways, Rasch dichotomous and partial credit analyses indicated similarly strong psychometric findings across models. Student outcome measures on the PSM6, scored both dichotomously and with partial credit, demonstrated strong, positive, significant correlation. Similar demographic patterns were noted regardless of scoring method. Both scoring methods produced similar results, suggesting that either would be appropriate to use with the PSM6.

Epistemic knowledge – a vital part of scientific literacy?

ABSTRACT The past century has seen a debate on what characterises a scientifically literate citizen. Originally, scientific literacy implied that a citizen should know the products of science but has grown to incorporate processes of science and aspects of the nature of science. Studies on students’ epistemic knowledge are rarer than ones on students’ content knowledge. However, PISA has made an innovative attempt to measure epistemic knowledge. The purpose of this study is to investigate qualitative features in student responses to epistemic items in PISA. Six constructed response items on epistemic knowledge from PISA 2018 Science were chosen. The student responses coded as ‘full credit’ were put aside and thematic analysis was used to analyse the remaining ‘no credit’ responses. The results show that it is common to focus on the scientific phenomenon (content knowledge) or the method itself (procedural knowledge) instead of epistemic knowledge. One feasible explanation is that epistemic knowledge is intellectually demanding; another is that epistemic knowledge is still overshadowed firstly by content knowledge and secondly by procedural knowledge. The fact that students tackle epistemic aspects of science as content or procedural knowledge suggests that teaching should more explicitly distinguish between these different aspects of scientific knowledge.