Multiple Matrix Sampling Research Articles

A practical guide to item bank calibration with multiple matrix sampling

When it is required to estimate item parameters of a large item bank, Multiple Matrix Sampling (MMS) design provides an efficient way while minimizing the test burden on students. The current study exemplifies how to calibrate a large item pool using MMS design for various purposes, such as developing a CAT administration. The purpose of the current study is to explain and provide an example of how to use MMS design for item bank calibration. Two functions of mirt package, mirt() and multipleGroup() were compared using real data. The results of the present study showed that the standard mirt() function is more practical and makes more precise estimations compared to the multipleGroup() function.

Is It Just a Game?

Abstract: Building on the long common history of board games and intelligence research, we developed a new deductive reasoning test based on the popular game Mastermind. The research questions of this registered report were: (a) Is a psychometrically sound measurement of the ability to solve Mastermind items possible (i.e., a reliable, uni-dimensional measurement with a good coverage of difficulty)? (b) Is the ability to solve Mastermind items substantially related to other measures of cognitive ability (i.e., matrix test, knowledge test) and need for cognition? (c) Can item difficulty be predicted by the number of colors, positions, premises, and a newly proposed entropy-based index? Based on the results of a pilot study, we developed 30 items and administered them to 351 participants in the preregistered main study using a multiple matrix sampling design. The deductive Mastermind test proved to be (a) a reliable and efficient measure of reasoning across a wide ability range, and (b) showed expectation-consistent patterns for the convergent and divergent measures. (c) The entropy-based index allowed for the prediction of item difficulty to a considerable degree ( R2 = .47). We discuss the ideas of information theory, including entropy, as constructing principles for the rational test development of reasoning tests.

Factor Structure and Measurement Invariance of the Self-discipline Model Using the Different-length Questionnaires: Application of Multiple Matrix Sampling

The purposes of this study were to test the structural validity and to test the parameters invariance of the self-discipline measurement model for good student citizenship among the models, using the data from the 1,047 complete questionnaires and the reducing length questionnaires with multiple matrix sampling technique. The sample size of this study was 1,047 bachelor's degree students selected by means of a multi-stage sampling technique. A set of 5 rating scale and 89-item questionnaires, divided into two versions, 44 and 45 each. The data were analyzed by confirmatory factor analysis and multiple group analysis. The research findings were as follows: 1) the model can be used to confirm the self-discipline construct validity of 4 components as responsibility, honesty, compliance and patience with ambition and intention, and 2) patterns and parameters of the model were varied between the models using the data from the full questionnaire and the data that reduced the length of the questionnaire. However, the measurement model is consistent with the empirical data in both cases.

Effects of Reducing the Length of the Questionnaire by Multiple Matrix Sampling on the Validity of Structural Equation Modeling for Factors Affecting Job Morale

The research was designed to examine the effects of question setting using different conditions into 10 sets on the validity of structural equation modeling for factors affecting job morale. The data was collected from 690 personnel working in regional Statistical Offices around Thailand by using cluster random sampling. The tool used in collecting data was the questionnaire with 95 items and 5 levels of rating scale. The discrimination value was between 0.244 and 0.860 and the reliability: α was from 0.699 to 0.900. Data analysis was conducted by the use of descriptive statistics and structural equation modeling by Mplus 7.4 program. The findings showed that the structural equation modeling with question setting in 3 sets of sub-questionnaires, cooperated with item sampling without replacement and non-fixing core items, conformed to the empirical data the most and that overall relationship between the structural model parameter and the model parameter from the complete questionnaire was high in a positive way.

Modeling Item Position Effects Using Generalized Linear Mixed Models

Item position effects can seriously bias analyses in educational measurement, especially when multiple matrix sampling designs are deployed. In such designs, item position effects may easily occur if not explicitly controlled for. Still, in practice it usually turns out to be rather difficult—or even impossible—to completely control for effects due to the position of items. The objectives of this article are to show how item position effects can be modeled using the linear logistic test model with additional error term (LLTM +ε) in the framework of generalized linear mixed models (GLMMs), to explore in a simulation study how well the LLTM +ε holds the nominal Type I risk threshold, to conduct power analysis for this model, and to examine the sensitivity of the LLTM +ε to designs that are not completely balanced concerning item position. Overall, the LLTM +ε proved suitable for modeling item position effects when a balanced design is used. With decreasing balance, the model tends to be more conservative in the sense that true item position effects are more unlikely to be detected. Implications for linking and equating procedures which use common items are discussed.

Item response theory in educational assessment and evaluation

Item response theory provides a useful and theoretically well-founded framework for educational measurement. It supports such activities as the construction of measurement instruments, linking and equating measurements, and evaluation of test bias and differential item functioning. It further provides underpinnings for item banking and flexible test administration designs, such as multiple matrix sampling, flexi-level testing, and computerized adaptive testing. First, a concise introduction to the principles of IRT models is given. The models discussed pertain to dichotomous items (items that are scored as either correct or incorrect) and polytomous items (items with partial credit scoring, such as most types of openended questions and performance assessments). Second, it is shown how an IRT measurement model can be enhanced with a structural model, such as, for instance, an analysis of variance model, to relate data from achievement and ability tests to students’ background variables, such as socio-economic status, intelligence or cultural capital, to school variables, and to features of the schooling system. Two applications are presented. The first one pertains to equating and linking of assessments, and the second one to a combination of an IRT measurement model and a multilevel linear model useful in school effectiveness research.

Split Questionnaire Design for Massive Surveys

Companies are conducting more and longer surveys than ever before. Massive questionnaires are pervasive in marketing practice. As an alternative to the heuristic methods that are currently used to split questionnaires, this study develops a methodology to design the split questionnaire in a way that minimizes information loss. Using estimates from a first wave or pilot study, the authors apply the modified Fedorov algorithm using the Kullback–Leibler distance as a design criterion to find the optimal splits. Their design criterion is based on a general mixed data model that accommodates continuous, rank-ordered, and discrete measurement scales. The optimal construction of the split questionnaire design is easy and fast. The authors use Markov chain Monte Carlo procedures to impute missing values that result from the design. They generate split questionnaire designs by selecting either entire blocks of questions (between-block design) or sets of questions in each block (within-block design). They compare the efficiency of split questionnaires generated with the proposed method with multiple matrix sampling designs, incomplete block designs, and a heuristic procedure, using synthetic and empirical Web survey data. The authors illustrate in a field study that as a result of reduced respondent burden, the quality of data using split questionnaire designs improves.

Split Questionnaire Design

Instead of the heuristic randomization methods to design split questionnaires that are currently used in applied and academic research, we develop a methodology to design the split questionnaire to minimize information loss. Because the number of possible questionnaire designs is exponential in the number of questions, we apply the Modified Federov algorithm, using Kullback Leibler Distance as a design criterion, to find the optimal splits. First of all, we illustrate the efficiency of the Modified Federov Algorithm on a small synthetic questionnaire, which enables the enumeration of all possible designs for comparison. Second, we compare the efficiency of split questionnaires generated with the proposed method to multiple matrix sampling (randomly generated designs) and a heuristic procedure based on principal components analysis, on synthetic and empirical data. We generate split questionnaire designs selecting either entire blocks of questions (between-block design) or sets of questions in each block (withinblock design). Finally, we illustrate that due to reduced respondent burden the quality of data using split designs increases, compared to a full questionnaire in a field study.

An Empirical Examination of a Modified Matrix Sampling Procedure as an Evaluation Tool for Grades 7 to 12 in a Midwestern School District

Multiple matrix sampling is a data collection technique that ensures accuracy and efficiency in group performance. It has been widely used in large-scale curriculum evaluation since the 1980s. However, the design does not always fully embrace the dynamics of local evaluation demands. The purpose of this study is to introduce a modified matrix sampling design to meet the specific need of a local school district to collect evaluation data. The most salient difference of the modified matrix sampling method in the present study lay in the incorporation of core items, while dividing the remainder of the items in the entire item pool into two instruments. The results from empirical examination of the modified matrix sampling design indicated that the approach is efficient at no extra cost for accuracy.

A Comparison of Item Sampling Plans in the Application of Multiple Matrix Sampling

This study used a Monte Carlo approach to investigate the effect of item sampling by item stratification on parameter estimation when applying multiple matrix sampling to achievement data. From the results of this study it was concluded that the item sampling method and sampling plan which is a practical compromise in terms of precision and sample size is one based on item stratification by item discrimination and a sampling plan with a moderate number of subtests. This sampling condition provides reasonable precision of the mean and variance estimates but requires only a moderately sized sample.

Parameter Estimation Using Multiple Matrix Sampling

AbstractMonte Carlo studies offer the opportunity to manipulate data sets with specified characteristics and to examine the generalizability of statistical procedures in ways that are not practical using actual (empirical) data. To determine whether computer-generated (simulated) data results accurately represent empirical data, this study replicated an investigation of the effects of item sampling plans in the application of multiple matrix sampling using both simulated and empirical data sets. Although results were similar, the empirical data results were more precise. This study suggests that, for some investigations, it may be important to confirm simulated study with empirical study.

Considerations and Techniques for the Analysis of NAEP Data

The special characteristics of the National Assessment data that affect the validity of conventional techniques of statistical analysis are considered. In contrast to the assumptions underlying standard methods of statistical analysis, the NAEP samples are obtained by means of a stratified multistage probability sampling design in which clusters of students are selected and in which certain subpopulations are sampled at a higher rate. The resulting samples have different statistical characteristics from those of a simple random sample. Analytic techniques that take the structure of the data into account are discussed and their properties explored. A second feature of the National Assessment data arises from the fact that multiple matrix sampling techniques are used for the assignment of subsets of the pool of exercises to subsamples of students. National Assessment accounts for the effect of the missing data on its scaling models through the use of multiple imputation procedures, creating vectors of “plausible values” for the scale scores. The use of plausible values and their effect on analysis are illustrated. The analyses are illustrated with results from the NAEP assessment of mathematics.

CONSIDERATIONS AND TECHNIQUES FOR THE ANALYSIS OF NAEP DATA

ABSTRACTThe special characteristics of the National Assessment data which affect the validity of conventional techniques of statistical analysis are considered. In contrast to the assumptions underlying standard methods of statistical analysis, the NAEP samples are obtained via a stratified multi‐stage probability sampling design in which clusters of students are selected and in which certain subpopulations are sampled at a higher rate. The resulting samples have different statistical characteristics from those of a simple random sample. Analytic techniques which take the structure of the data into account are discussed and their properties explored. A second feature of the National Assessment data arises from the fact that multiple matrix sampling techniques are used for the assignment of subsets of the pool of exercises to subsamples of students. National Assessment accounts for the effect of the missing data on its scaling models through the use of multiple imputation procedures; creating vectors of “plausible values” for the scale‐scores. The use of plausible values and their effect on analysis are illustrated. The analyses are illustrated with results from the NAEP assessments of mathematics.

The Use of Multiple Matrix Sampling for Survey Research

Multiple matrix sampling procedures can be employed to improve survey research when the results of matrix sampling are equivalent to those obtained by the traditional census testing approach. This study examined the use of multiple matrix sampling as a strategy for the collection of data and compared rates of response when subgroups of items were administered as opposed to an entire instrument. In addition, the study investigated whether responses were equivalent in the two sampling procedures and whether bias was present. The results indicate that multiple matrix sampling is a viable and reasonable procedure to use when a mail survey questionnaire consists of a large number of pages and/or items.

A Note on the Application of Multiple Matrix Sampling to Standard Setting

In many of the methods currently proposed for standard setting, all experts are asked to judge all items, and the standard is taken as the mean of their judgments. When resources are limited, gathering the judgments of all experts in a single group can become impractical. Multiple matrix sampling (MMS) provides an alternative. This paper applies MMS to a variation on Angoff's method (1971) of standard setting. A pool of 36 experts and 190 items were divided randomly into 5 groups, and estimates of borderline examinee performance were acquired. Results indicated some variability in the cutting scores produced by the individual groups, but the variance components were reasonably well estimated. The standard error of the cutting score was very small, and the width of the 90% confidence interval around it was only 1.3 items. The reliability of the final cutting score was.98

Estimating test norms via multiple matrix sampling.

Estimating test norms via multiple matrix sampling.

A COMPARISON OF EXAMINEE SAMPLING AND MULTIPLE MATRIX SAMPLING IN TEST DEVELOPMENT

For the purpose of obtaining data to use in test development, multiple matrix sampling (MMS) plans were compared to examinee sampling plans. Data were simulated for examinees, sampled from a population with a normal distribution of ability, responding to items selected from an item universe. Three item universes were considered: one that would produce a normal distribution of test scores, one a moderately platykurtic distribution, and one a very platykurtic distribution. When comparing sampling plans, total numbers of observations were held constant. No differences were found among plans in estimating item difficulty. Examinee sampling produced better estimates of item discrimination, test reliability, and test validity. As total number of observations increased, estimates improved considerably, especially for those MMS plans with larger subtest sizes. Larger numbers of observations were needed for tests designed to produce a normal distribution of test scores. With an adequate number of observations, MMS is seen as an alternative to examinee sampling in test development.

Efficient estimation for a multiple matrix sample design

A generalized least squares method is proposed for estimation in multiple matrix sampling designs. It is shown that this provides efficient estimates, is flexible, and makes fewer assumptions than other procedures.

An Item Response Model for the Estimation of Demographic Effects

Item sampling procedures employed in many assessment studies are designed so that each respondent answers a small number of items in each of a large number of skill areas. If the item population contains several items designed to measure an underlying variable such as computation skill, it may be desirable to fit an item response model to the data. In studies that employ multiple matrix sampling there are not enough answers from each individual to employ such a model. To circumvent this problem, a model is formulated on the assumption that individual level variability appears as independent error within the cells of the cross classification of manifest demographic variables such as sex and race. This model is successfully fit to a scale of items, involving addition of fractions, from the National Assessment Study.

A Matrix Sampling Study of the Children's Adaptive Behavior Scale

A multiple matrix sampling procedure was used to obtain local norms on the Children's Adaptive Behavior Scale (CABS). Matrix sampling is briefly explained and then exemplified in the method section. A sample of 158 children was drawn from a population of 784 fourth graders in a local Midwestern school system. The estimates based on the local sample were compared with the standardization data from the preliminary national sample. Of the five CABS subtests, two were significantly different. Differences between variances of the local and national samples were also observed. The standard deviations of the lo cal sample were more restricted. The implications of the findings are discussed as they relate to the assessment of children's adaptive behavior functioning.