Random Item Selection Research Articles

On the effect of items measuring different factors across individuals on item inter-correlations

Even when a single factor is to be measured, it may occur in the context of blind random item selection that individuals work on items that are not based on exactly the same factor. Therefore, we explore the consequences of presenting items from different populations measuring different factors across and within individuals. We found that item inter-correlations can be substantial in the total population of individuals even when – in subpopulations of individuals – items are drawn from populations based on different, even uncorrelated factors. In order to address this challenge for convergent validity, we propose a method that helps to detect whether the correlation between items is due to the same common factor measured by the items across all individuals. Based on the analytical results and results from a simulation study, we provide recommendations for the detection of subpopulations of individuals responding to items from different item populations.

Probabilistic Evaluation of the Exploration–Exploitation Balance during the Search, Using the Swap Operator, for Nonlinear Bijective S-Boxes, Resistant to Power Attacks

During the search for S-boxes resistant to Power Attacks, the S-box space has recently been divided into Hamming Weight classes, according to its theoretical resistance to these attacks using the metric variance of the confusion coefficient. This partition allows for reducing the size of the search space. The swap operator is frequently used when searching with a random selection of items to be exchanged. In this work, the theoretical probability of changing Hamming Weight class of the S-box is calculated when the swap operator is applied randomly in a permutation. The precision of these probabilities is confirmed experimentally. Its limit and a recursive formula are theoretically proved. It is shown that this operator changes classes with high probability, which favors the exploration of the Hamming Weight class of S-boxes space but dramatically reduces the exploitation within classes. These results are generalized, showing that the probability of moving within the same class is substantially reduced by applying two swaps. Based on these results, it is proposed to modify/improve the use of the swap operator, replacing its random application with the appropriate selection of the elements to be exchanged, which allows taking control of the balance between exploration and exploitation. The calculated probabilities show that the random application of the swap operator is inappropriate during the search for nonlinear S-boxes resistant to Power Attacks since the exploration may be inappropriate when the class is resistant to Differential Power Attack. It would be more convenient to search for nonlinear S-boxes within the class. This result provides new knowledge about the influence of this operator in the balance exploration–exploitation. It constitutes a valuable tool to improve the design of future algorithms for searching S-boxes with good cryptography properties. In a probabilistic way, our main theoretical result characterizes the influence of the swap operator in the exploration–exploitation balance during the search for S-boxes resistant to Power Attacks in the Hamming Weight class space. The main practical contribution consists of proposing modifications to the swap operator to control this balance better.

Clinical Correlates and Implications of the Reliability of the Frailty Index in the Canadian Longitudinal Study on Aging.

Deficit accumulation frailty indices (FIs) are widely used to characterize frailty. FIs vary in number and composition of items; the impact of this variation on reliability and clinical applicability is unknown. We simulated 12 000 studies using a set of 70 candidate deficits in 12 080 community-dwelling participants 65 years and older. For each study, we varied the number (5, 10, 15, 25, 35, 45) and composition (random selection) of items defining the FI and calculated descriptive and predictive estimates: frailty score, prevalence, frailty cutoff, mortality odds ratio, predicted probability of mortality for FI = 0.28 (prevalence threshold), and FI cutoff predicting 10% mortality over the follow-up. We summarized the estimates' medians and spreads (0.025-0.975 quantiles) by number of items and calculated intraclass correlation coefficients (ICCs). Medians of frailty scores were 0.11-0.12 with decreasing spreads from 0.04-0.24 to 0.10-0.14 for 5-item and 45-item FIs. The median cutoffs identifying 15% as frail was 0.19-0.20 and stable; the spreads decreased with more items. However, medians and spreads for the prevalence of frailty (median: 11%-3%), mortality odds ratio (median: 1.24-2.19), predicted probability of mortality (median: 8%-17%), and FI cutoff predicting 10% mortality (median: 0.38-0.20) varied markedly. ICC increased from 0.19 (5-item FIs) to 0.84 (45-item FIs). Variability in the number and composition of items of individual FIs strongly influences their reliability. Estimates using FIs may not be sufficiently stable for generalizing results or direct application. We propose avenues to improve the development, reporting, and interpretation of FIs.

Liner programming method to construct equated item sets for implementation of periodical computer-based testing of Korean medical licensing examination.

PurposeThis study aimed to identify the best way of developing equivalent item sets and to propose a stable and effective management plan for periodical licensing examinations. MethodsFive pre-equated item sets were developed based on the predicted correct answer rate of each item using linear programming. These pre-equated item sets were compared to the ones that were developed with a random item selection method based on the actual correct answer rate (ACAR) and difficulty from item response theory (IRT). The results with and without common items were also compared in the same way. ACAR and the IRT difficulty were used to determine whether there was a significant difference between the pre-equating conditions. ResultsThere was a statistically significant difference in IRT difficulty among the results from different pre-equated conditions. The predicted correct answer rate was divided using 2 or 3 difficulty categories, and the ACAR and IRT difficulty parameters of the 5 item sets were equally constructed. Comparing the item set conditions with and without common items, including common items did not make a significant contribution to the equating of the 5 item sets. ConclusionThis study suggested that the linear programming method is applicable to construct equated-item sets that reflect each content area. The suggested best method to construct equated item sets is to divide the predicted correct answer rate using 2 or 3 difficulty categories, regardless of common items. If pre-equated item sets are required to construct a test based on the actual data, several methods should be considered by simulation studies to determine which is optimal before administering a real test.

Structure–reactivity modeling using mixture-based representation of chemical reactions

We describe a novel approach of reaction representation as a combination of two mixtures: a mixture of reactants and a mixture of products. In turn, each mixture can be encoded using an earlier reported approach involving simplex descriptors (SiRMS). The feature vector representing these two mixtures results from either concatenated product and reactant descriptors or the difference between descriptors of products and reactants. This reaction representation doesn't need an explicit labeling of a reaction center. The rigorous "product-out" cross-validation (CV) strategy has been suggested. Unlike the naïve "reaction-out" CV approach based on a random selection of items, the proposed one provides with more realistic estimation of prediction accuracy for reactions resulting in novel products. The new methodology has been applied to model rate constants of E2 reactions. It has been demonstrated that the use of the fragment control domain applicability approach significantly increases prediction accuracy of the models. The models obtained with new "mixture" approach performed better than those required either explicit (Condensed Graph of Reaction) or implicit (reaction fingerprints) reaction center labeling.

Multidimensional Computerized Adaptive Testing for Classifying Examinees With Within-Dimensionality.

A classification method is presented for adaptive classification testing with a multidimensional item response theory (IRT) model in which items are intended to measure multiple traits, that is, within-dimensionality. The reference composite is used with the sequential probability ratio test (SPRT) to make decisions and decide whether testing can be stopped before reaching the maximum test length. Item-selection methods are provided that maximize the determinant of the information matrix at the cutoff point or at the projected ability estimate. A simulation study illustrates the efficiency and effectiveness of the classification method. Simulations were run with the new item-selection methods, random item selection, and maximization of the determinant of the information matrix at the ability estimate. The study also showed that the SPRT with multidimensional IRT has the same characteristics as the SPRT with unidimensional IRT and results in more accurate classifications than the latter when used for multidimensional data.

Selection of Common Items as an Unrecognized Source of Variability in Test Equating: A Bootstrap Approximation Assuming Random Sampling of Common Items

The standard error of equating quantifies the variability in the estimation of an equating function. Because common items for deriving equated scores are treated as fixed, the only source of variability typically considered arises from the estimation of common-item parameters from responses of samples of examinees. Use of alternative, equally appropriate anchor sets results in different equating transformations, even when standard errors of common-item parameter estimates are negligible. A bootstrap approximation for quantifying the variability due to common-item selection is derived for a statewide assessment assuming random selection of common items from a large hypothetical item pool. The standard error of equating and the error due to common-item selection constitute only a small fraction of the variability for individual examinee scores. For group means, other sources of error shrink as sample size increases; unaffected by sample size, error due to common-item selection may become the dominant source of variability.

Classification accuracy and consistency of computerized adaptive testing

In this article, four item selection methods in computerized adaptive testing are examined in terms of classification accuracy and consistency, including two popular heuristics for constraint management, the maximum priority index (MPI) method and the weighted deviation modeling method, as well as the widely known maximum Fisher information method and randomized item selection as baselines. Results suggest that the MPI method is able to meet constraints and keep test overlap rate low. Among the four methods, it is the only one that manages to produce parallel forms in terms of content coverage and, consequently, the only method to which the idea of classification consistency applies. With tests as short as 12 items, the MPI method does fairly well in classifying examinees accurately and consistently. Its performance improves with longer tests. The effects of number of decision categories and cut score locations are also examined. Recommendations are made in the Discussion section.

Comparing single-pool and multiple-pool designs regarding test security in computerized testing

This article compares the use of single- and multiple-item pools with respect to test security against item sharing among some examinees in computerized testing. A simulation study was conducted to make a comparison among different pool designs using the item selection method of maximum item information with the Sympson-Hetter exposure control and content balance. The results from the simulation study indicate that two-pool designs have a better degree of resistance to item sharing than do the single-pool design in terms of measurement precision in ability estimation. This article further characterizes the conditions under which employing a multiple-pool design is better than using a single, whole pool in terms of minimizing the number of compromised items encountered by examinees under a randomized item selection method. Although no current computerized testing program endorses the randomized item selection method, the results derived in this study can shed some light on item pool designs regarding test security for all item selection algorithms, especially those that try to equalize or balance item exposure rates by employing a randomized item selection method locally, such as the a-stratified-with-b-blocking method.

Mapping the distributions and quantifying the labelling intensities of cell compartments by immunoelectron microscopy: progress towards a coherent set of methods

An important tool in cell biology is the combination of immunogold labelling and transmission electron microscopy (TEM) by which target molecules (e.g. antigens) are bound specifically to affinity markers (primary antibodies) and then detected and localised with visualisation probes (e.g. colloidal gold particles bound to protein A). Gold particles are electron-dense, punctate and available in different sizes whilst TEM provides high-resolution images of particles and cell compartments. By virtue of these properties, the combination can be used also to quantify one or more defined targets in cell compartments. During the past decade, new ways of quantifying gold labelling within cells have been devised. Their efficiency and validity rely on sound principles of specimen sampling, event counting and inferential statistics. These include random selection of items at each sampling stage (e.g. specimen blocks, thin sections, microscopical fields), stereological analysis of cell ultrastructure, unbiased particle counting and statistical evaluation of a suitable null hypothesis (no difference in the intensity or pattern of labelling between compartments or groups of cells). The following approaches are possible: (i) A target molecule can be tested for preferential labelling by mapping the localisation of gold particles across a set of compartments. (ii) Data from wild-type and knockdown/knockout control cells can be used to correct raw gold particle counts, estimate specific labelling densities and then test for preferential labeling. (iii) The same antigen can be mapped in two or more groups of cells to test whether there are experimental shifts in compartment labelling patterns. (iv) A variant of this approach uses more than one size of gold particle to test whether or not different antigens colocalise in one or more compartments. (v) In studies involving antigen translocation, absolute numbers of gold particles can be mapped over compartments at specific positions within polarised, oriented or dividing cells. Here, the current state of the art is reviewed and approaches are illustrated with virtual datasets.

Developing brief fatigue short forms calibrated to a common mathematical metric: is content-balancing important?

There are clinical and research settings in which concerns about respondent burden make the use of longer self-report measures impractical. Though computer adaptive testing provides an efficient strategy for measuring patient reported outcomes, the requirement of a computer interface makes it impractical for some settings. This study evaluated how well brief short forms, constructed from a longer measure of patient reported fatigue, reproduced scores on the full measure. When the items of an item bank are calibrated using an item response theory model, it is assumed that the items are fungible units. Theoretically, there should be no advantage to balancing the content coverage of the items. We compared short forms developed using a random item selection process to short forms developed with consideration of the items relation to subdomains of fatigue (ie, physical and cognitive fatigue). Scores on short forms developed using content balancing more successfully predicted full item bank scores than did scores on short forms developed by random selection of items.

Stratified and Maximum Information Item Selection Procedures in Computer Adaptive Testing

In this study we evaluated and compared three item selection procedures: the maximum Fisher information procedure (F), the a‐stratified multistage computer adaptive testing (CAT) (STR), and a refined stratification procedure that allows more items to be selected from the high a strata and fewer items from the low a strata (USTR), along with completely random item selection (RAN). The comparisons were with respect to error variances, reliability of ability estimates and item usage through CATs simulated under nine test conditions of various practical constraints and item selection space. The results showed that F had an apparent precision advantage over STR and USTR under unconstrained item selection, but with very poor item usage. USTR reduced error variances for STR under various conditions, with small compromises in item usage. Compared to F, USTR enhanced item usage while achieving comparable precision in ability estimates; it achieved a precision level similar to F with improved item usage when items were selected under exposure control and with limited item selection space. The results provide implications for choosing an appropriate item selection procedure in applied settings.

A probability-based sampling approach for the analysis of drug seizures composed of multiple containers of either cocaine, heroin, or Cannabis

A probability-based analytical sampling approach for seized containers of cocaine, Cannabis, or heroin, to answer questions of both content weight and identity, is described. It utilizes the Student's t distribution, and, because of the lack of normality in studied populations, the power of the Central Limit Theorem with samples of size 20 to calculate the mean net weights of multiple item drug seizures. Populations studied ranged between 50 and 1200 units. Identity determination is based on chemical testing and sampling using the hypergeometric distribution fit to a program macro – created by the European Network of Forensic Science Institutes (ENFSI) Drugs Working Group. Formal random item selection is effected through use of an Excel ®-generated list of random numbers. Included, because of their impact on actual practice, are discussions of admissibility, sufficiency of proof, method validation, and harmony with the guidelines of international standardizing bodies.

When Cognitive Diagnosis Meets Computerized Adaptive Testing: CD-CAT

Computerized adaptive testing (CAT) is a mode of testing which enables more efficient and accurate recovery of one or more latent traits. Traditionally, CAT is built upon Item Response Theory (IRT) models that assume unidimensionality. However, the problem of how to build CAT upon latent class models (LCM) has not been investigated until recently, when Tatsuoka (J. R. Stat. Soc., Ser. C, Appl. Stat. 51:337–350, 2002) and Tatsuoka and Ferguson (J. R. Stat., Ser. B 65:143–157, 2003) established a general theorem on the asymptotically optimal sequential selection of experiments to classify finite, partially ordered sets. Xu, Chang, and Douglas (Paper presented at the annual meeting of National Council on Measurement in Education, Montreal, Canada, 2003) then tested two heuristics in a simulation study based on Tatsuoka’s theoretical work in the context of computerized adaptive testing. One of the heuristics was developed based on Kullback–Leibler information, and the other based on Shannon entropy. In this paper, we showcase the application of the optimal sequential selection methodology in item selection of CAT that is built upon cognitive diagnostic models. Two new heuristics are proposed, and are compared against the randomized item selection method and the two heuristics investigated in Xu et al. (Paper presented at the annual meeting of National Council on Measurement in Education, Montreal, Canada, 2003). Finally, we show the connection between the Kullback–Leibler-information-based approaches and the Shannon-entropy-based approach, as well as the connection between algorithms built upon LCM and those built upon IRT models.

A Conditional Exposure Control Method for Multidimensional Adaptive Testing

In computerized adaptive testing (CAT), ensuring the security of test items is a crucial practical consideration. A common approach to reducing item theft is to define maximum item exposure rates, i.e., to limit the proportion of examinees to whom a given item can be administered. Numerous methods for controlling exposure rates have been proposed for tests employing the unidimensional 3‐PL model. The present article explores the issues associated with controlling exposure rates when a multidimensional item response theory (MIRT) model is utilized and exposure rates must be controlled conditional upon ability. This situation is complicated by the exponentially increasing number of possible ability values in multiple dimensions. The article introduces a new procedure, called the generalized Stocking‐Lewis method, that controls the exposure rate for students of comparable ability as well as with respect to the overall population. A realistic simulation set compares the new method with three other approaches: Kullback‐Leibler information with no exposure control, Kullback‐Leibler information with unconditional Sympson‐Hetter exposure control, and random item selection.

Constraint-Weighted a-Stratification for Computerized Adaptive Testing With Nonstatistical Constraints

a-stratification is a method that utilizes items with small discrimination (a) parameters early in an exam and those with higher a values when more is learned about the ability parameter. It can achieve much better item usage than the maximum information criterion (MIC). To make a-stratification more practical and more widely applicable, a method for weighting the item selection process in a-stratification as a means of satisfying multiple test constraints is proposed. This method is studied in simulation against an analogous method without stratification as well as a-stratification using descending-rather than ascending- a procedures. In addition, a variation of a-stratification that allows for unbalanced usage of a parameters is included in the study to examine the trade-off between efficiency and exposure control. Finally, MIC and randomized item selection are included as baseline measures. Results indicate that the weighting mechanism successfully addresses the constraints, that stratification helps to a great extent balancing exposure rates, and that the ascending- a design improves measurement precision.

Severity of Organized Item Theft in Computerized Adaptive Testing: A Simulation Study

Criteria had been proposed for assessing the severity of possible test security violations for computerized tests with high-stakes outcomes. However, these criteria resulted from theoretical derivations that assumed uniformly randomized item selection. This study investigated potential damage caused by organized item theft in computerized adaptive testing (CAT) for two realistic item selection methods, maximum item information and a-stratified with content blocking, using the randomized method as a baseline for comparison. Damage caused by organized item theft was evaluated by the number of compromised items each examinee could encounter and the impact of the compromised items on examinees' ability estimates. Severity of test security violation was assessed under self-organized and organized item theft simulation scenarios. Results indicated that though item theft could cause severe damage to CAT with either item selection method, the maximum item information method was more vulnerable to the organized item theft simulation than was the a-stratified method.

SEVERITY OF ORGANIZED ITEM THEFT IN COMPUTERIZED ADAPTIVE TESTING: AN EMPIRICAL STUDY

Chang and Zhang (2002, 2003) proposed several baseline criteria for assessing the severity of possible test security violations for computerized tests with high-stakes outcomes. However, these criteria were obtained from theoretical derivations that assumed uniformly randomized item selection. The current study investigated potential damage caused by organized item theft in computerized adaptive testing (CAT) for two more realistic item selection methods, the maximum item information and the a-stratified, while using the randomized method as a baseline for comparison. The results of the study indicated that the damage could be very severe, especially when the thieves took the test in the early stage of utilization of an item pool. Among the three CAT methods examined in this study, the maximum item information method with Sympson-Hetter exposure control was most vulnerable to organized item theft.

Computer adaptive testing improved accuracy and precision of scores over random item selection in a physical functioning item bank

Backgrond and Objective Measuring physical functioning (PF) within and across postacute settings is critical for monitoring outcomes of rehabilitation; however, most current instruments lack sufficient breadth and feasibility for widespread use. Computer adaptive testing (CAT), in which item selection is tailored to the individual patient, holds promise for reducing response burden, yet maintaining measurement precision. We calibrated a PF item bank via item response theory (IRT), administered items with a post hoc CAT design, and determined whether CAT would improve accuracy and precision of score estimates over random item selection. Methods 1,041 adults were interviewed during postacute care rehabilitation episodes in either hospital or community settings. Responses for 124 PF items were calibrated using IRT methods to create a PF item bank. We examined the accuracy and precision of CAT-based scores compared to a random selection of items. Results CAT-based scores had higher correlations with the IRT-criterion scores, especially with short tests, and resulted in narrower confidence intervals than scores based on a random selection of items; gains, as expected, were especially large for low and high performing adults. Conclusion The CAT design may have important precision and efficiency advantages for point-of-care functional assessment in rehabilitation practice settings.

Optimal and Nonoptimal Computer-Based Test Designs for Making Pass–Fail Decisions

Now that many credentialing exams are being routinely administered by computer, new computer-based test designs, along with item response theory models, are being aggressively researched to identify specific designs that can increase the decision consistency and accuracy of pass–fail decisions. The purpose of this study was to investigate the impact of optimal and nonoptimal multistage test (MST) designs, linear parallel-form test designs (LPFT), and computer adaptive test (CAT) designs on the decision consistency and accuracy of pass–fail decisions. Realistic testing situations matching those of one of the large credentialing agencies were simulated to increase the generalizability of the findings. The conclusions were clear: (a) With the LPFTs, matching test information functions (TIFs) to the mean of the proficiency distribution produced slightly better results than matching them to the passing score; (b) all of the test designs worked better than test construction using random selection of items, subject to content constraints only; (c) CAT performed better than the other test designs; and (d) if matching a TIF to the passing score, the MST design produced a bit better results than the LPFT design. If an argument for the MST design is to be made, it can be made on the basis of slight improvements over the LPFT design and better expected item bank utilization, candidate preference, and the potential for improved diagnostic feedback, compared with the feedback that is possible with fixed linear test forms.