Multidimensional Adaptive Testing Research Articles

혼합형 문항으로 구성된 단계적 적응검사에서 교차 라우팅 방법의 효과 탐색

Multistage testing among several adaptive testing types has been more frequently applied into
 practice because it can accurately and efficiently estimate abilities with relatively short tests. However,
 it has been also criticized due to its low level of adaptability. To address this issue, the intersectional
 routing method was proposed and its effectiveness has been partly confirmed. This study aims to
 explore the effectiveness of the intersectional routing method with mixed-format MST by applying the
 bi-factor MIRT model, and the UIRT model, extending previous researches. Based upon the results of
 this study, when the module length was short (8 items), and the correlation between sections was
 high (0.9), the accuracy of ability estimates using the intersectional routing was higher than the
 typical multistage testing design regardless of the IRT model. The application of the intersectional
 routing technique decreased the differences of ability estimate accuracy in accordance with panel
 designs. Considering the high inter-correlation values between sections in actual tests (ranging from 0.6
 to 0.9), the application of the intersectional routing method into multistage testing could be a choice
 for broadening options in test designs and obtaining more accurate ability estimates. The results also
 indicate the necessity of selecting an appropriate IRT model based on test characteristics in
 multidimensional adaptive testing situations.

The Automated Test Assembly and Routing Rule for Multistage Adaptive Testing with Multidimensional Item Response Theory

AbstractDesigning a multidimensional adaptive test (M‐MST) based on a multidimensional item response theory (MIRT) model is critical to make full use of the advantages of both MST and MIRT in implementing multidimensional assessments. This study proposed two types of automated test assembly (ATA) algorithms and one set of routing rules that can facilitate the development of an M‐MST. Different M‐MST designs were developed based on the proposed ATA algorithms and routing rules and were evaluated through two sets of simulation studies. Study 1 used simulated item banks and considered a variety of testing factors, and results from which can inform us the theoretical performance of the proposed M‐MST designs. Study 2 was designed based on a real multidimensional assessment. In addition, a MCAT was simulated as a baseline design to demonstrate the advantage of the proposed M‐MST design in each study. Our simulation results indicate that the proposed ATA algorithms and routing rule can generate M‐MSTs with a good‐quality control and the same or even better ability estimation results than the MCAT given the same condition. This demonstrates the advantage of using M‐MST for multidimensional assessment especially for the one that needs to satisfy many nonstatistical constraints.

Impact of Violation of the Missing-at-Random Assumption on Full-Information Maximum Likelihood Method in Multidimensional Adaptive Testing.

The full-information maximum likelihood (FIML) method makes it possible to estimate and analyze structural equation models (SEM) even when data are partially missing, enabling incomplete data to contribute to model estimation. The cornerstone of FIML is the missing-at-random (MAR) assumption. In (unidimensional) computerized adaptive testing (CAT), unselected items (i.e., responses that are not observed) remain at random even though selected items (i.e., responses that are observed) have been associated with a test taker’s latent trait that is being measured. In multidimensional adaptive testing (MAT), however, the missingness in the response data partially depends on the unobserved data because items are selected based on various types of information including the covariance among latent traits. This eventually may lead to violations of MAR. This study aimed to evaluate the potential impact such a violation of MAR in MAT could have on FIML estimation performance. The results showed an increase in estimation errors in item parameter estimation when the MAT response data were used, and differences in the level of the impact depending on how items loaded on multiple latent traits.

Application of a multidimensional computerized adaptive test for a Clinical Dementia Rating Scale through computer-aided techniques

BackgroundWith the increasingly rapid growth of the elderly population, individuals aged 65 years and above now compose 14% of Taiwanese citizens, thereby making Taiwanese society an aged society. A leading factor that affects the elderly population is dementia. A method of precisely and efficiently examining patients with dementia through multidimensional computer adaptive testing (MCAT) to accurately determine the patients’ stage of dementia needs to be developed. This study aimed to develop online MCAT that family members can use on their own computers, tablets, or smartphones to predict the extent of dementia for patients responding to the Clinical Dementia Rating (CDR) instrument.MethodsThe CDR was applied to 366 outpatients in a hospital in Taiwan. MCAT was employed with parameters for items across eight dimensions, and responses were simulated to compare the efficiency and precision between MCAT and non-adaptive testing (NAT). The number of items saved and the estimated person measures was compared between the results of MCAT and NAT, respectively.ResultsMCAT yielded substantially more precise measurements and was considerably more efficient than NAT. MCAT achieved 20.19% (= [53 − 42.3]/53) saving in item length when the measurement differences were less than 5%. Pearson correlation coefficients were highly consistent among the eight domains. The cut-off points for the overall measures were − 1.4, − 0.4, 0.4, and 1.4 logits, which was equivalent to 20% for each portion in percentile scores. Substantially fewer items were answered through MCAT than through NAT without compromising the precision of MCAT.ConclusionsDeveloping a website that family members can use on their own computers, tablets, and smartphones to help them perform online screening and prediction of dementia in older adults is useful and manageable.

Testlet-Based Multidimensional Adaptive Testing.

Multidimensional adaptive testing (MAT) is a highly efficient method for the simultaneous measurement of several latent traits. Currently, no psychometrically sound approach is available for the use of MAT in testlet-based tests. Testlets are sets of items sharing a common stimulus such as a graph or a text. They are frequently used in large operational testing programs like TOEFL, PISA, PIRLS, or NAEP. To make MAT accessible for such testing programs, we present a novel combination of MAT with a multidimensional generalization of the random effects testlet model (MAT-MTIRT). MAT-MTIRT compared to non-adaptive testing is examined for several combinations of testlet effect variances (0.0, 0.5, 1.0, and 1.5) and testlet sizes (3, 6, and 9 items) with a simulation study considering three ability dimensions with simple loading structure. MAT-MTIRT outperformed non-adaptive testing regarding the measurement precision of the ability estimates. Further, the measurement precision decreased when testlet effect variances and testlet sizes increased. The suggested combination of the MTIRT model therefore provides a solution to the substantial problems of testlet-based tests while keeping the length of the test within an acceptable range.

Heuristic Constraint Management Methods in Multidimensional Adaptive Testing.

Although multidimensional adaptive testing (MAT) has been proven to be highly advantageous with regard to measurement efficiency when several highly correlated dimensions are measured, there are few operational assessments that use MAT. This may be due to issues of constraint management, which is more complex in MAT than it is in unidimensional adaptive testing. Very few studies have examined the performance of existing constraint management methods (CMMs) in MAT. The present article focuses on the effectiveness of two promising heuristic CMMs in MAT for varying levels of imposed constraints and for various correlations between the measured dimensions. Through a simulation study, the multidimensional maximum priority index (MMPI) and multidimensional weighted penalty model (MWPM), as an extension of the weighted penalty model, are examined with regard to measurement precision and constraint violations. The results show that both CMMs are capable of addressing complex constraints in MAT. However, measurement precision losses were found to differ between the MMPI and MWPM. While the MMPI appears to be more suitable for use in assessment situations involving few to a moderate number of constraints, the MWPM should be used when numerous constraints are involved.

R Package MAT: Simulation of Multidimensional Adaptive Testing for Dichotomous IRT Models.

Multidimensional adaptive testing (MAT) is ideal for testing situations that require the estimation of two or more distinct, albeit correlated latent abilities. MAT offers a number of advantages over unidimensional adaptive testing, such as more adequate coverage of content (Segall, 1996), increased measurement efficiency for variable-length tests (Segall, 1996), and increased measurement precision for fixed-length tests (Wang, Chen, & Cheng, 2004). Simulation of MAT on both empirical and generated data is useful for developing computerized adaptive testing (CAT) systems and for teaching the principles of MAT.

Hypothetical Use of Multidimensional Adaptive Testing for the Assessment of Student Achievement in the Programme for International Student Assessment

The usefulness of multidimensional adaptive testing (MAT) for the assessment of student literacy in the Programme for International Student Assessment (PISA) was examined within a real data simulation study. The responses of N = 14,624 students who participated in the PISA assessments of the years 2000, 2003, and 2006 in Germany were used to simulate MAT with different restrictions (unrestricted, treatment of link items, treatment of open items, content balance, unitwise item selection, all restrictions). Compared with conventional testing based on the booklet design of PISA 2006, unrestricted MAT increases measurement efficiency by 74% and reduces the average number of presented items from 55 to 26 without a loss in measurement precision. The incorporation of restrictions reduces the advantages of MAT. MAT is recommended for the assessment of newly introduced constructs but not for the assessment of the literacy domains in PISA.

Measuring physical function in children with airway support: A pilot study using computer adaptive testing

Objective: To assess the responsiveness, examine the scoring range and determine the efficiency of a multidimensional computer adaptive testing version of the Pediatric Evaluation of Disability Inventory (PEDI-MCAT) for children admitted to inpatient pulmonary rehabilitation.Methods: The PEDI-MCAT was completed by clinician report for 30 infants and children. Mean self-care and mobility admission scores were compared with discharge scores for the total group and two diagnostic sub-groups (prematurity and congenital/neurological conditions). The scoring range of the mobility and self-care scales was examined to determine placement of the scores along the overall PEDI-MCAT scale. Efficiency was determined using an internal clock and average number of items required for score generation.Results: Mean changes for the total group and both sub-groups were significant for both self-care and mobility, except for the prematurity group's mobility scores. Effect sizes were small-to-moderate. Scores for both groups were at the low end of the scoring ranges. Average time to complete the PEDI-MCAT was 1.57 minutes. Average number of items administered was nine for self-care and 11 for mobility.Conclusion: The PEDI-MCAT was responsive to change in physical function, although only low-ability items were needed. The PEDI-MCAT can potentially minimize clinician burden in inpatient settings.

Multidimensional adaptive testing in educational and psychological measurement: Current state and future challenges

The paper gives an overview of multidimensional adaptive testing (MAT) and evaluates its applicability in educational and psychological testing. The approach of Segall (1996) is described as a general framework for MAT. The main advantage of MAT is its capability to increase measurement efficiency. In simulation studies conceptualizing situations typical to large scale assessments, the number of presented items was reduced by MAT by about 30–50% compared to unidimensional adaptive testing and by about 70% compared to fixed item testing holding measurement precision constant. Empirical results underline these findings. Before MAT is used routinely some open questions should be answered first. After that, MAT represents a very promising approach to highly efficient simultaneous testing of multiple competencies.

Multidimensional Adaptive Testing with Optimal Design Criteria for Item Selection

Several criteria from the optimal design literature are examined for use with item selection in multidimensional adaptive testing. In particular, it is examined what criteria are appropriate for adaptive testing in which all abilities are intentional, some should be considered as a nuisance, or the interest is in the testing of a composite of the abilities. Both the theoretical analyses and the studies of simulated data in this paper suggest that the criteria of A-optimality and D-optimality lead to the most accurate estimates when all abilities are intentional, with the former slightly outperforming the latter. The criterion of E-optimality showed occasional erratic behavior for this case of adaptive testing, and its use is not recommended. If some of the abilities are nuisances, application of the criterion of As-optimality (or Ds-optimality), which focuses on the subset of intentional abilities is recommended. For the measurement of a linear combination of abilities, the criterion of c-optimality yielded the best results. The preferences of each of these criteria for items with specific patterns of parameter values was also assessed. It was found that the criteria differed mainly in their preferences of items with different patterns of values for their discrimination parameters.

Adaptive Models of Psychological Testing

Psychological tests with a conventional format present the same set of items to each test taker. Such tests have been popular because they are easy to administer in paper-andpencil form to large groups of test takers at the same time. When testing became computerized in the early 1990s, the initial applications had the same linear format, though it did not take too long for computer-based testing to become adaptive. In an adaptive format, the computer updates the score of the test taker each time (s)he produces a response to an item and automatically adapts the choice of the next item to the last score. Statistically, adaptive testing requires comparability of test scores across different selections of items, and by the time computer-based testing began to mature, itemresponse theory had already become fully developed and was able to provide just that. As argued in the opening article of this special issue, these first applications in fact returned to the same principles of testing that any proctor involved in the testing of individual subjects followed before testing became fully standardized: immediate judgment of the quality of the responses and avoidance of questions that appear too easy or too difficult for the subject. The advantages of adaptive testing are a much shorter test length to realize the same level of score precision as for a linear test, uniform precision over a much wider range of the ability being tested, more motivated test takers because they are challenged by the items but do not find them unduly difficult, and higher security, there being no paperand-pencil forms to be compromised (albeit less security if the test can be taken continuously in high-stakes applications). In addition, adaptive testing shares the benefits of any type of computer-based testing, such as immediate scoring, the possibility of using innovative item types, and the electronic monitoring of test takers as well as bookkeeping of their performances. These gains are only realized, however, when the testing agent is willing to invest in extensive item writing and calibration as well as the adequate software and hardware required to run the test. The motivation for this special issue of the Zeitschrift fur Psychologie / Journal of Psychology was to demonstrate the benefits of adaptive models for psychological testing and to explore the issues associated with their implementation. Until recently, the main technological advances and applications of computerized adaptive testing lay in the domain of educational testing. This development was surprising because standardized testing actually started in psychology. Also, item-response theory – a necessary tool for running adaptive testing programs – was developed in psychometrics, one of the earliest subdisciplines to emerge in psychology. The outline of the issue is as follows: the opening article, “Some New Developments in Adaptive Testing Technology” (van der Linden), highlights the historic roots of adaptive testing and introduces its main principles. It then reviews some of the newer developments in its technology, such as constrained adaptive testing, rule-based generation of item pools for adaptive testing, multidimensional adaptive testing, optimal sequencing of adaptive test batteries, and the use of response times to improve adaptive testing. The next two articles, “Computerized Adaptive Testing of Personality Traits” by A. Michiel Hol, Harry C.M. Vorst, and Gideon J. Mellenbergh; and “Transitioning from Fixed-Length Questionnaires to Computer Adaptive Testing” by Otto B. Walter and Heinz Holling, show the advantages of, and the efforts involved in, creating an adaptive version of a conventional linear instrument. The example used by Hol et al. is a Dutch version of the dominance scale of Gough and Heilbrun’s Adjective Check List (ACL) with Likert response scales with five categories. Walter and Holling transformed a German translation of the Interpersonal Competence Questionnaire (ICQ) into an adaptive instrument and evaluated the impact of the transition on the test scores. The articles “Computer Adaptive-Attribute Testing: A New Approach to Cognitive Diagnostic Assessment” by Mark J. Gierl and Jiawen Zhou; and “ICAT: An Adaptive Testing Procedure to Allow the Identification of Idiosyncratic Knowledge Patterns” by G. Gage Kingsbury and Ronald L. Houser explore the possibilities of an adaptive testing format for cognitive diagnosis. Relying heavily on developments in

Some New Developments in Adaptive Testing Technology

In an ironic twist of history, modern psychological testing has returned to an adaptive format quite common when testing was not yet standardized. Important stimuli to the renewed interest in adaptive testing have been the development of item-response theory in psychometrics, which models the responses on test items using separate parameters for the items and test takers, and the use of computers in test administration, which enables us to estimate the parameter for a test taker and select the items in real time. This article reviews a selection from the latest developments in the technology of adaptive testing, such as constrained adaptive item selection, adaptive testing using rule-based item generation, multidimensional adaptive testing, adaptive use of test batteries, and the use of response times in adaptive testing.

Implementation and Measurement Efficiency of Multidimensional Computerized Adaptive Testing

Multidimensional adaptive testing (MAT) procedures are proposed for the measurement of several latent traits by a single examination. Bayesian latent trait estimation and adaptive item selection are derived. Simulations were conducted to compare the measurement efficiency of MAT with those of unidimensional adaptive testing and random administration. The results showed that the higher the correlation between latent traits, the more latent traits there were, and the more scoring levels there were in the items, the more efficient MAT was than the other two procedures. For tests containing multidimensional items, only MAT is applicable, whereas unidimensional adaptive testing is not. Issues in implementing MAT are discussed.

Multidimensional adaptive testing with constraints on test content

The case of adaptive testing under a multidimensional response model with large numbers of constraints on the content of the test is addressed. The items in the test are selected using a shadow test approach. The 0–1 linear programming model that assembles the shadow tests maximizes posterior expected Kullback-Leibler information in the test. The procedure is illustrated for five different cases of multidimensionality. These cases differ in (a) the numbers of ability dimensions that are intentional or should be considered as “nuisance dimensions” and (b) whether the test should or should not display a simple structure with respect to the intentional ability dimensions.

Multidimensional adaptive testing for mental health problems in primary care.

Efficient and accurate instruments for assessing child psychopathology are increasingly important in clinical practice and research. For example, screening in primary care settings can identify children and adolescents with disorders that may otherwise go undetected. However, primary care offices are notorious for the brevity of visits and screening must not burden patients or staff with long questionnaires. One solution is to shorten assessment instruments, but dropping questions typically makes an instrument less accurate. An alternative is adaptive testing, in which a computer selects the items to be asked of a patient based on the patient's previous responses. This research used a simulation to test a child mental health screen based on this technology. Using half of a large sample of data, a computerized version was developed of the Pediatric Symptom Checklist (PSC), a parental-report psychosocial problem screen. With the unused data, a simulation was conducted to determine whether the Adaptive PSC can reproduce the results of the full PSC with greater efficiency. PSCs were completed by parents on 21,150 children seen in a national sample of primary care practices. Four latent psychosocial problem dimensions were identified through factor analysis: internalizing problems, externalizing problems, attention problems, and school problems. A simulated adaptive test measuring these traits asked an average of 11.6 questions per patient, and asked five or fewer questions for 49% of the sample. There was high agreement between the adaptive test and the full (35-item) PSC: only 1.3% of screening decisions were discordant (kappa = 0.93). This agreement was higher than that obtained using a comparable length (12-item) short-form PSC (3.2% of decisions discordant; kappa = 0.84). Multidimensional adaptive testing may be an accurate and efficient technology for screening for mental health problems in primary care settings.

Multidimensional adaptive testing

Maximum likelihood and Bayesian procedures for item selection and scoring of multidimensional adaptive tests are presented. A demonstration using simulated response data illustrates that multidimensional adaptive testing (MAT) can provide equal or higher reliabilities with about one-third fewer items than are required by one-dimensional adaptive testing (OAT). Furthermore, holding test-length constant across the MAT and OAT approaches, substantial improvements in reliability can be obtained from multidimensional assessment. A number of issues relating to the operational use of multidimensional adaptive testing are discussed.