Automatic Item Generation Research Articles

Evaluating the impact of depth cue salience in working three-dimensional mental rotation tasks by means of psychometric experiments

The gender difference in three-dimensional mental rotation is well documented in the literature. In this article we combined automatic item generation, (quasi-)experimental research designs and item response theory models of change measurement to evaluate the effect of the ability to extract the depth information conveyed in the two-dimensional illustrations on (1) females and males test performance and (2) subsequent cognitive processes called upon in the course of working three-dimensional mental rotation tasks. On the basis of prior studies we hypothesized that experimentally increasing the salience of depth cues would lead to an improvement in respondents' test performance that is more pronounced in females than males. This hypothesis was tested and confirmed in the first study. In the second study we extended the experimental design to take respondents' mental transformation strategies into account. We hypothesized that increasing the salience of depth cues would also lead to changes in respondents' mental transformation strategies. One of the main findings of this study was an increased use of more efficient mental transformation strategies that was more pronounced in females than males. Furthermore, the induced shift to more effective mental transformation strategies accounted for a sizeable portion of the reduction of the gender gap. The implications of the results of both studies are discussed.

The Impact on Ability Estimates of Predicted Parameters from Cognitively Designed Items in a Computerized Adaptive Testing Environment

Automatic item generation has become a promising area in recent studies. In automatic item generation, items with targeted psychometric properties are generated during testing. The feasibility of automatic item generation lies in the fact that items are generated from a set of observable item stimulus features, which are mapped onto the cognitive variables underlying the item solution and are calibrated through cognitive psychometric models. Parameters of a generated item can then be predicted from the specific combination of the calibrated item stimulus features in the item. Predicted item parameters, compared to those calibrated from empirical data, involve more complex sources of uncertainty. Although the relationship between sufficiency of the cognitive model of item solving and the adequacy of item parameter prediction can be theoretically justified, the degree to which such predicted parameters impact various aspects of testing, however, is an empirical question and needs to be explored. This paper investigated the impact of predicted item parameters on ability estimates in a computerized adaptive testing environment, based on abstract reasoning test (ART) items which were generated using cognitive design system approach (Embretson, 1998). The item bank contained 150 items with two sets of item difficulties, of which one is predicted from the item design features and the other is calibrated from sample data. Each of the 263 subjects participated in the study received two subtests, of which one was based on predicted parameters, the other calibrated parameters. The item bank was split into two parallel halves based on predicted item parameters to prevent items in the bank from repeat administrations within subjects. Subjects were randomly assigned into one of the four testing procedures resulting from the combinations between parameter types (predicted versus calibrated) and item bank (first half versus second half). Results of the study showed a clear regression-to-the-mean effect of the predicted item parameters, compare to its counterpart of calibrated item parameters. Inward biases of ability estimates from subtest using predicted item parameters were observed when ability estimates were compared across different subtests within subjects. Compared to its counterpart using calibrated parameters, standard errors of ability estimates were larger for those from the subtest using predicted item parameters in the mid-range of the scale, where regression-to-the-mean effect of the predicted item parameters is minimal, and were smaller in the rest of the scale, possibly due to joint impact of increased uncertainty of predicted item parameters, estimation biases and limitation of the item bank at various levels of ability scale. When ability was estimated from the same subtest using different types of item parameters, very high correlation (.995) were obtained and no biases were observed throughout almost the entire scale. Standard errors of ability estimates were larger for predicted parameters yet the differences were small.

Evaluating the contribution of different item features to the effect size of the gender difference in three-dimensional mental rotation using automatic item generation

In complex three-dimensional mental rotation tasks males have been reported to score up to one standard deviation higher than females. However, this effect size estimate could be compromised by the presence of gender bias at the item level, which calls the validity of purely quantitative performance comparisons into question. We hypothesized that the effect of gender bias at the level of distinct item design features could lead to either an over- or underestimation of reported effect sizes of the gender difference in three-dimensional mental rotation. Using automatic item generation we conducted a series of psychometric experiments in which we independently manipulated one out of four different item design features that have exhibited a gender bias in the previous studies (study 1). This was done in a between-subjects design. The results indicated that gender bias caused by item design features linked to the perceptual stadium of mental rotation led to an overestimation of the effect size of the gender difference while item design features associated with the encoding and transformational stadium resulted in an underestimation of the effect size of the gender difference. In study 2 we tested the hypothesis that the gender difference still remains while controlling for the item design features causing gender bias. The results suggest that a significant portion of the gender difference may be attributable to perceptual and encoding processes involved in mental rotation.

Combining automatic item generation and experimental designs to investigate the contribution of cognitive components to the gender difference in mental rotation

Marked gender differences in three-dimensional mental rotation have been broadly reported in the literature in the last few decades. Various theoretical models and accounts were used to explain the observed differences. Within the framework of linking item design features of mental rotation tasks to cognitive component processes associated with them we combined automatic item generation, (quasi-)experimental research designs and think aloud protocols to test several hypotheses: in study 1 the main focus was on gender differences regarding (1) encoding- and mental transformation strategies and (2) the ability to maintain a stable mental representation while transforming it. The results indicated that females are more prone to either lose track or to lose their internal mental representations during transformation process. Furthermore, they also tended to use more error-prone mental transformation strategies. The second study focused on the evaluation of the hypotheses that the observed gender difference can be accounted for by individual differences in the ability to extract the depth information conveyed in the illustrations of the object. Based on previous studies we gradually manipulated the salience of depth cues in a psychometric experiment. The results partially confirmed our initial hypotheses: depending on the degree of salience of the depth information females either benefitted more, or to an equal extent than males from our experimental manipulations. The implications of the results of both studies are discussed.

Automatic item generation of probability word problems

Mathematical word problems represent a common item format for assessing student competencies. Automatic item generation (AIG) is an effective way of constructing many items with predictable difficulties, based on a set of predefined task parameters. The current study presents a framework for the automatic generation of probability word problems based on templates that allow for the generation of word problems involving different topics from probability theory. It was tested in a pilot study with N = 146 German university students. The items show a good fit to the Rasch model. Item difficulties can be explained by the Linear Logistic Test Model (LLTM) and by the random-effects LLTM. The practical implications of these findings for future test development in the assessment of probability competencies are also discussed.

Investigating the ‘g’-saturation of various stratum-two factors using automatic item generation

Even though researchers agree on the hierarchical structure of intelligence there is considerable disagreement on the g-saturation of the lower stratum-two factors. In this article it is argued that the mixed evidence in the research literature can be at least partially attributed to the construct representation of the individual tests used to measure the stratum-two factors. In the study described here two top-down approaches to automatic item generation were used to build construct representation directly into the items. This enabled a clearer substantive interpretation of the shared commonalities extracted by the various stratum-two factors and helped to rule out alternative, but mathematically equivalent, structural models based on independent empirical evidence. This approach was used to investigate the g-saturation of five stratum-two factors using a sample of 240 male and female respondents. The results support the assumption that fluid intelligence exhibits the highest g-saturation.

Using psychometric technology in educational assessment: The case of a schema-based isomorphic approach to the automatic generation of quantitative reasoning items

This article deals with the investigation of the psychometric quality and constructs validity of algebra word problems generated by means of a schema-based version of the automatic min–max approach. Based on review of the research literature in algebra word problem solving and automatic item generation this new approach is introduced as a theory-based top–down method of automatic item generation featuring a quality control framework aimed to minimize the construct unrelated variance in the item parameters. The first study deals with the evaluation of an initial set of items. The results are replicated in the second study using a larger item set which also allows the investigation of the construct representation of the generated item. Since construct unrelated variance components (e.g. reading comprehension) have been controlled for in the item generation phase the results revealed some interesting insights into the cognitive processes of the actual mathematization phase of algebra word problem solving. The third study investigated the nomothetic span is using hierarchical confirmatory factor analysis. The results argue for the convergent and discriminant validity of the automatically generated items. Taken together, the results indicate that the automatic generation of construct valid algebra word problems at a high psychometric level is viable. The discussion is thus concerned with the implications of this new approach to item generation for theory development and evaluation as well as practical benefits for educational assessment and the development of intelligent tutoring systems.

Automatic Generation of Quantitative Reasoning Items

This paper deals with three studies on the computer-based, automatic generation of algebra word problems. The cognitive psychology based generative/quality control frameworks of the item generator are presented. In Study I the quality control framework is empirically tested using a first set of automatically generated items. Study II replicates the findings of Study I using a larger set of automatically generated algebra word problems. Study III deals with the generative framework of the item generator by testing construct validity aspects of the item generator produced items. Using nine Rasch-homogeneous subscales of the new intelligence structure battery (INSBAT, Hornke et al., 2004 ), a hierarchical confirmatory factor analysis is reported, which provides first evidence of convergent as well as divergent validity of the automatically generated items. The end of the paper discusses possible advantages of automatic item generation in general ranging from test security issues and the possibility of a more precise psychological assessment to mass testing and economical questions of test construction.

Automatic Generation of Rasch-Calibrated Items: Figural Matrices Test GEOM and Endless-Loops Test EC

The future of test construction for certain psychological ability domains that can be analyzed well in a structured manner may lie—at the very least for reasons of test security—in the field of automatic item generation. In this context, a question that has not been explicitly addressed is whether it is possible to embed an item response theory (IRT) based psychometric quality control procedure directly into the process of automatic item generation. Research in this area was conducted using 2 item generators (for the 2 domains of reasoning and spatial ability) that were developed and based on relevant models of cognitive psychology. During the course of the 4 studies reported here, those parts of the generators that check for possible violations of psychometric quality ("constraints") were improved. The main findings indicate that quality control procedures can be embedded in automatic item generators depending on (a) the degree to which the domain to be measured can be structured; (b) item-specific, content-based analyses; and (c) the degree to which the constraints can be implemented in software. Furthermore, beyond the global check of given model fit via IRT, the content-based analysis of items may be valuable in terms of finding such item properties that may lead to violations of psychometric quality.

Calibrating Item Families and Summarizing the Results Using Family Expected Response Functions

Item families, which are groups of related items, are becoming increasingly popular in complex educational assessments. For example, in automatic item generation (AIG) systems, a test may consist of multiple items generated from each of a number of item models. Item calibration or scoring for such an assessment requires fitting models that can take into account the dependence structure inherent among the items that belong to the same item family. Glas and van der Linden (2001) suggest a Bayesian hierarchical model to analyze data involving item families with multiple-choice items. We fit the model using the Markov Chain Monte Carlo (MCMC) algorithm, introduce the family expected response function (FERF) as a way to summarize the probability of a correct response to an item randomly generated from an item family, and suggest a way to estimate the FERFs. This work is thus a step towards creating a tool that can save significant amount of resources in educational testing, by allowing proper analysis and summarization of data from tests involving item families.

AN APPLICATION OF A BAYESIAN HIERARCHICAL MODEL FOR ITEM FAMILY CALIBRATION

ABSTRACTItem families, which are groups of items related to each other in some way, are increasingly used in complex educational assessments. For example, in automatic item generation (AIG) systems, a test may consist of multiple items generated from a number of item models. Item calibration or scoring for such an assessment requires fitting models that can take into account the dependence structure inherent among the items that belong to the same item family. Glas and van der Linden (2001) suggest a Bayesian hierarchical model to analyze data involving item families. We fit that hierarchical model using the Markov chain Monte Carlo (MCMC) algorithm. Formulating the MCMC algorithm provides little additional difficulty (compared to fitting a simple item response theory model) even with the additional complexity in the form of hierarchy in the model. We show that the model can take into account the dependence structure inherent among the items and hence is an improvement over the models currently used in similar situations. We introduce the notion of the family expected response function (FERF) as a way to summarize the probability of a correct response to an item randomly generated from an item family and suggest a way to estimate the FERFs. Our work is a step towards creating a tool that can save significant amount of resources for tests with item families, because calibrating only the item families might be enough rather than calibrating each item in the families separately.