Expected Response Functions Research Articles

Scoring Graphical Responses in TIMSS 2019 Using Artificial Neural Networks.

Automated scoring of free drawings or images as responses has yet to be used in large-scale assessments of student achievement. In this study, we propose artificial neural networks to classify these types of graphical responses from a TIMSS 2019 item. We are comparing classification accuracy of convolutional and feed-forward approaches. Our results show that convolutional neural networks (CNNs) outperform feed-forward neural networks in both loss and accuracy. The CNN models classified up to 97.53% of the image responses into the appropriate scoring category, which is comparable to, if not more accurate, than typical human raters. These findings were further strengthened by the observation that the most accurate CNN models correctly classified some image responses that had been incorrectly scored by the human raters. As an additional innovation, we outline a method to select human-rated responses for the training sample based on an application of the expected response function derived from item response theory. This paper argues that CNN-based automated scoring of image responses is a highly accurate procedure that could potentially replace the workload and cost of second human raters for international large-scale assessments (ILSAs), while improving the validity and comparability of scoring complex constructed-response items.

Fast ion energy distribution from third harmonic radio frequency heating measured with a single crystal diamond detector at the Joint European Torus.

Neutron spectroscopy measurements with a single crystal diamond detector have been carried out at JET, for the first time in an experiment aimed at accelerating deuterons to MeV energies with radio frequency heating at the third harmonic. Data are interpreted by means of the expected response function of the detector and are used to extract parameters of the highly non-Maxwellian distribution function generated in this scenario. A comparison with observations using a time of flight and liquid scintillator neutron spectrometers is also presented. The results demonstrate the capability of diamond detectors to contribute to fast ion physics studies at JET and are of more general relevance in view of the application of such detectors for spectroscopy measurements in the neutron camera of next step tokamak devices.

COMPARING DIFFERENT APPROACHES OF BIAS CORRECTION FOR ABILITY ESTIMATION IN IRT MODELS

ABSTRACTThe method of maximum‐likelihood is typically applied to item response theory (IRT) models when the ability parameter is estimated while conditioning on the true item parameters. In practice, the item parameters are unknown and need to be estimated first from a calibration sample. Lewis (1985) and Zhang and Lu (2007) proposed the expected response functions (ERFs) and the corrected weighted‐likelihood estimator (CWLE), respectively, to take into account the uncertainty regarding item parameters for purposes of ability estimation. In this paper, we investigate the performance of ERFs and of the CWLE in different situations, such as various test lengths and levels of measurement error in item parameter estimation. Our empirical results indicate that ERFs can cause the bias in ability estimation to fall within [‐0.2, 0.2] for all conditions, whereas the CWLE can effectively reduce the bias in ability estimation provided that it has a good foundation to start from.

A Beta Item Response Model for Continuous Bounded Responses

An item response model is proposed for the analysis of continuous response formats in an item response theory (IRT) framework. With such formats, respondents are asked to report their response as a mark on a fixed-length graphical segment whose ends are labeled with extreme responses. An interpolation process is proposed as the response mechanism in this situation, from which the authors show that a beta distribution on the response naturally follows. The two natural parameters of the beta are expressed as monotonic functions of person and item distance on some latent continuum. It is shown that a logistic expected response function results from a simple choice for these monotonic functions. A joint maximum likelihood estimation procedure is presented that performs well in a simulation study. An application on mood items is also presented, with very good fit properties.

Calibrating Item Families and Summarizing the Results Using Family Expected Response Functions

Item families, which are groups of related items, are becoming increasingly popular in complex educational assessments. For example, in automatic item generation (AIG) systems, a test may consist of multiple items generated from each of a number of item models. Item calibration or scoring for such an assessment requires fitting models that can take into account the dependence structure inherent among the items that belong to the same item family. Glas and van der Linden (2001) suggest a Bayesian hierarchical model to analyze data involving item families with multiple-choice items. We fit the model using the Markov Chain Monte Carlo (MCMC) algorithm, introduce the family expected response function (FERF) as a way to summarize the probability of a correct response to an item randomly generated from an item family, and suggest a way to estimate the FERFs. This work is thus a step towards creating a tool that can save significant amount of resources in educational testing, by allowing proper analysis and summarization of data from tests involving item families.

Shape-invariant modeling of circadian rhythms with random effects and smoothing spline ANOVA decompositions.

Medical studies often collect physiological and/or psychological measurements over time from multiple subjects, to study dynamics such as circadian rhythms. Under the assumption that the expected response functions of all subjects are the same after shift and scale transformations, shape-invariant models have been applied to analyze this kind of data. The shift and scale parameters provide efficient and interpretable data summaries, while the common shape function is usually modeled nonparametrically, to provide flexibility. However, due to the deterministic nature of the shift and scale parameters, potential correlations within a subject are ignored. Furthermore, the shape of the common function may depend on other factors, such as disease. In this article, we propose shape-invariant mixed effects models. A second-stage model with fixed and random effects is used to model individual shift and scale parameters. A second-stage smoothing spline ANOVA model is used to study potential covariate effects on the common shape function. We apply our methods to a real data set to investigate disease effects on circadian rhythms of cortisol, a hormone that is affected by stress. We find that patients with Cushing's syndrome lost circadian rhythms and their 24-hour means were elevated to very high levels. Patients with major depression had the same circadian shape and phases as normal subjects. However, their 24-hour mean levels were elevated and amplitudes were dampened for some patients.

AN APPLICATION OF A BAYESIAN HIERARCHICAL MODEL FOR ITEM FAMILY CALIBRATION

ABSTRACTItem families, which are groups of items related to each other in some way, are increasingly used in complex educational assessments. For example, in automatic item generation (AIG) systems, a test may consist of multiple items generated from a number of item models. Item calibration or scoring for such an assessment requires fitting models that can take into account the dependence structure inherent among the items that belong to the same item family. Glas and van der Linden (2001) suggest a Bayesian hierarchical model to analyze data involving item families. We fit that hierarchical model using the Markov chain Monte Carlo (MCMC) algorithm. Formulating the MCMC algorithm provides little additional difficulty (compared to fitting a simple item response theory model) even with the additional complexity in the form of hierarchy in the model. We show that the model can take into account the dependence structure inherent among the items and hence is an improvement over the models currently used in similar situations. We introduce the notion of the family expected response function (FERF) as a way to summarize the probability of a correct response to an item randomly generated from an item family and suggest a way to estimate the FERFs. Our work is a step towards creating a tool that can save significant amount of resources for tests with item families, because calibrating only the item families might be enough rather than calibrating each item in the families separately.

AN ANALYSIS OF SIMILARITIES IN ITEM FUNCTIONING WITHIN ANTONYM AND ANALOGY VARIANT FAMILIES

ABSTRACTThis study examined the potential advantages of using item variants — sets of related test items based on a common template or schema — to offset the increased item‐production demands associated with continuous computer‐based testing in high‐stakes assessments. Empirical results showed that the calibrated difficulty parameters of analogy and antonym item variants that had been used in operational administrations of the Graduate Record Examinations (GRE®) General Test were very similar. Results of a simulation study that considered precision losses within a linear testing framework projected precision losses of less than 10% for estimating examinee abilities from responses to variants when using expected response functions based just on variant‐family information, compared to using true item response theory parameters for individual items.

An Expected Response Function Approach to Graphical Differential Item Functioning

In this paper a new approach to graphical differential item functioning (DIF) is offered. The methodology is based on a sampling‐theory approach to expected response functions (Lewis, 1985; Mislevy, Wingersky, & Sheehan, 1994). Essentially error in item calibrations is modeled explicitly, and repeated samples are taken from the posterior distributions of the item parameters. Sampled parameter values are used to estimate the posterior distribution of the difference in item characteristic curves (ICCs)for two groups. A point‐wise expectation is taken as an estimate of the true difference between the ICCs, and the sampled‐difference functions indicate uncertainty in the estimate. Tbe approach is applied to a set of pretest items, and the results are compared to traditional Mantel‐Haenszel DIF statistics. The expected‐response‐function approach is contrasted with Pashley's (1992) graphical DIF approach.

DEALING WITH UNCERTAINTY ABOUT ITEM PARAMETERS: EXPECTED RESPONSE FUNCTIONS

ABSTRACTIt is a common practice in item response theory (IRT) to treat estimates of item parameters, say B, as if they were the known, true quantities, B. However, ignoring the uncertainty associated with item parameters can lead to biases and over‐confidence in subsequent inferences such as ability estimation, especially when item‐calibration samples are small. This paper demonstrates how to incorporate uncertainty about B with Lewis's “expected response functions” (ERFs), pointwise expected values of item response conditional on examinee proficiency averaged over posterior distributions of item parameters. This paper presents ERFs, outlines procedures for computing them and using them in practical work, and gives an illustration with data from the National Assessment of Educational Progress. Advantages of approximating ERFs response curves with members of familiar parametric families of IRT curves are noted.

Log contrast models for experiments with mixtures

SUMMARY A new form of expected response function involving log contrasts of the proportions is introduced for experiments with mixtures. The advantages and disadvantages of log contrast models are discussed and illustrated in applications. In particular, the parameters of the model are related to certain useful hypotheses about mixtures and some relevant contrasts are identified.

An Improved Definition of Synergistic and Antagonistic Effects

Presently accepted methods for calculating the expected response for combinations of agents such as herbicides or other growth regulators from the responses of individual agents can lead to unacceptable results. This may be true even if responses within the linear region of the individual response curves are used in such calculations. A solution to this problem is to assign parameters to response curves of individual agents and to define the expected response function for combinations of agents by the weighted algebraic means of these parameters.