Innovative Item Types Research Articles

Examining Position Effects on Students’ Ability and Test-Taking Speed in the TIMSS 2019 Problem-Solving and Inquiry Tasks: A Structural Equation Modeling Approach

Position effects occur when changes in item positions on a test impact the test outcomes (e.g., item parameters or test scores). Previous studies found that position effects may vary by the testing context and conditions and thus affect each test-taker differently. With the increasing adoption of digital assessments involving innovative item types that are focused on problem-solving skills, it is also essential to study position effects within this context. This study incorporates item-level scores and screen-level response time data from the Trends in International Mathematics and Science Study (TIMSS) 2019 problem-solving and inquiry tasks for Grade 4 students to examine position effects on students’ ability and test-taking speed. This dataset included responses from 27,682 students from 36 countries. A structural equation modeling approach was employed to model ability and test-taking speed within the same model. The results indicated a small but statistically significant booklet effect on students’ overall mathematics and science ability. The impact of block position changes within the booklets seemed to be greater than the impact of a reordering of subjects tested in the two sessions. The results also showed that when an item block was placed earlier in a test session, students spent more time on the items and performed better. The implications of these findings are discussed.

Measuring social and emotional skills in elementary students: Development of self-report Likert, situational judgment test, and forced choice items

As social and emotional learning (SEL) continues to gain popularity, the need for high-quality social and emotional skill assessments also increases. We conducted two studies to develop and validate items to measure social and emotional skills in third, fourth, and fifth grade students. The Big Five personality framework served as an assessment framework for image-enhanced Likert items, situational judgment test items, and forced choice items. Results from Study 1 (n = 1047) provided promising reliability and validity evidence, as well as concrete recommendations for item revisions. Study 2 (n = 826) was conducted with a revised item pool and demonstrated improved reliability and validity. Taken together, results provided initial support that social and emotional skills can be validly and reliably measured in elementary-aged students using innovative item types.

The Prevalence of Item Construction Flaws in Medical School Examinations and Innovative Recommendations for Improvement

The purpose of this study was to calculate the prevalence and nature of item construction flaws within one large medical school and to identify several innovative approaches that may serve as potential remedies for these problems. Results indicated that approximately one in five items contained a construction flaw, with the overwhelming majority of flaws involving poor quality distractors. A series of innovative recommendations are presented, including modern psychometric analytical techniques to more thoroughly inspect data, item manipulation techniques, and the use of innovative item types that may alleviate the need for distractors altogether.

Assessing Intercultural Competence in Higher Education: Existing Research and Future Directions

AbstractThe modern wave of globalization has created a demand for increased intercultural competence (ICC) in college graduates who will soon enter the 21st‐century workforce. Despite the wide attention to the concepts and assessment of ICC, few assessments meet the standards for a next‐generation assessment in areas of construct clarity, innovative item types, response processes, and validity evidence. The objectives of this report are to identify current conceptualizations of ICC, review existing assessments and their validity evidence, propose a new framework for a next‐generation ICC assessment, and discuss key assessment considerations. To summarize, we found the current state of the literature to be murky in terms of the clarity of the ICC construct. Definitions of the construct vary considerably as to whether it is a trait, skill, or performance outcome. In addition, current measurements of ICC overly rely on self‐report methods, which have a number of flaws that result in less than optimal assessment. In this paper, we propose a new framework based on a model of the social thinking process developed by Grossman and colleagues that describes the knowledge, skills, and abilities that promote success in complex social situations. From this social process model, as well as Earley and Peterson's definition of ICC (a person's capability to gather, interpret, and act upon these radically different cues to function effectively across cultural settings or in a multicultural situation), three stages are developed: approach, analyze, and act. Guided by this framework, we discuss assessment considerations such as innovative task types and multiple response formats to help translate the framework to an assessment of ICC.

Comparison of the reliability of scoring methods of multiple-response items, matching items, and sequencing items

Multiple-response items, sequencing items, and matching items are three innovative item types often included in systems for computer-based assessment that offer the benefit of polytomous scoring and the possibility to measure partial knowledge. In the present study, different scoring methods of these three item types were compared. Based on the assumption that different response patterns to these item types represent different knowledge levels, these knowledge levels are described. Features of different scoring methods were studied to select the scoring methods included in this study. Subsequently, a probability distribution of scoring results for each knowledge level was derived and computed. Based on classical test theory, a measure for the reliability of the different scoring methods on the level of a single item was derived. To compare the results of the scoring methods selected, reliabilities were computed for several distributions of knowledge levels in a population. For a multiple-response item, when an examinee must select all the right options, the dichotomous scoring method resulted in higher reliabilities than scoring the response patterns polytomously. For matching items and for multiple-response items, when an examinee is asked to select fewer options than the total number of right options given, polytomous scoring methods gave higher reliabilities than the dichotomous scoring method. Simple polytomous scoring by counting the selected right options or relations is recommended instead of more complex polytomous scoring methods, for instance, using a correction for wrong answers or a so-called "floor". The results of scoring sequencing items were not as conclusive as for the other two item types explored.

11 Computer-based competence tests in the national educational panel study: The challenge of mode effects

Computerized competence tests promise a variety of advantages compared to paper–pencil delivered tests, for instance, increased test security, more information about test takers and the test-taking process, instant scoring, and immediate feedback. Moreover, new innovative item types can be administered to broaden the test content. Three benefits should be particularly emphasized for the assessment of cognitive competencies in the German National Educational Panel Study. First, reductions of test time can be obtained through the higher measurement efficiency of adaptive tests. Second, computerized testing is expected to enhance standardization and to increase test takers’ interest in completing the test. Third, Internet-based assessment offers the opportunity to deliver tests to spatially distanced test takers. However, before we can exploit these opportunities, we have to study the equivalence between different test administrations in order to maintain comparability of test scores and to ensure the validity of score interpretations. In this chapter, we shall describe a theoretical framework of mode effects and discuss various properties of test administrations. We shall relate the resulting equivalence criteria to the specific settings of the National Educational Panel Study in which (a) the usage of computerized competence tests is being prepared for upcoming assessments, and (b) tests for different grades and age groups are being designed to assess competence development over the life span.

Adaptive Models of Psychological Testing

Psychological tests with a conventional format present the same set of items to each test taker. Such tests have been popular because they are easy to administer in paper-andpencil form to large groups of test takers at the same time. When testing became computerized in the early 1990s, the initial applications had the same linear format, though it did not take too long for computer-based testing to become adaptive. In an adaptive format, the computer updates the score of the test taker each time (s)he produces a response to an item and automatically adapts the choice of the next item to the last score. Statistically, adaptive testing requires comparability of test scores across different selections of items, and by the time computer-based testing began to mature, itemresponse theory had already become fully developed and was able to provide just that. As argued in the opening article of this special issue, these first applications in fact returned to the same principles of testing that any proctor involved in the testing of individual subjects followed before testing became fully standardized: immediate judgment of the quality of the responses and avoidance of questions that appear too easy or too difficult for the subject. The advantages of adaptive testing are a much shorter test length to realize the same level of score precision as for a linear test, uniform precision over a much wider range of the ability being tested, more motivated test takers because they are challenged by the items but do not find them unduly difficult, and higher security, there being no paperand-pencil forms to be compromised (albeit less security if the test can be taken continuously in high-stakes applications). In addition, adaptive testing shares the benefits of any type of computer-based testing, such as immediate scoring, the possibility of using innovative item types, and the electronic monitoring of test takers as well as bookkeeping of their performances. These gains are only realized, however, when the testing agent is willing to invest in extensive item writing and calibration as well as the adequate software and hardware required to run the test. The motivation for this special issue of the Zeitschrift fur Psychologie / Journal of Psychology was to demonstrate the benefits of adaptive models for psychological testing and to explore the issues associated with their implementation. Until recently, the main technological advances and applications of computerized adaptive testing lay in the domain of educational testing. This development was surprising because standardized testing actually started in psychology. Also, item-response theory – a necessary tool for running adaptive testing programs – was developed in psychometrics, one of the earliest subdisciplines to emerge in psychology. The outline of the issue is as follows: the opening article, “Some New Developments in Adaptive Testing Technology” (van der Linden), highlights the historic roots of adaptive testing and introduces its main principles. It then reviews some of the newer developments in its technology, such as constrained adaptive testing, rule-based generation of item pools for adaptive testing, multidimensional adaptive testing, optimal sequencing of adaptive test batteries, and the use of response times to improve adaptive testing. The next two articles, “Computerized Adaptive Testing of Personality Traits” by A. Michiel Hol, Harry C.M. Vorst, and Gideon J. Mellenbergh; and “Transitioning from Fixed-Length Questionnaires to Computer Adaptive Testing” by Otto B. Walter and Heinz Holling, show the advantages of, and the efforts involved in, creating an adaptive version of a conventional linear instrument. The example used by Hol et al. is a Dutch version of the dominance scale of Gough and Heilbrun’s Adjective Check List (ACL) with Likert response scales with five categories. Walter and Holling transformed a German translation of the Interpersonal Competence Questionnaire (ICQ) into an adaptive instrument and evaluated the impact of the transition on the test scores. The articles “Computer Adaptive-Attribute Testing: A New Approach to Cognitive Diagnostic Assessment” by Mark J. Gierl and Jiawen Zhou; and “ICAT: An Adaptive Testing Procedure to Allow the Identification of Idiosyncratic Knowledge Patterns” by G. Gage Kingsbury and Ronald L. Houser explore the possibilities of an adaptive testing format for cognitive diagnosis. Relying heavily on developments in

Measurement Efficiency of Innovative Item Formats in Computer‐Based Testing

The psychometric literature provides little empirical evaluation of examinee test data to assess essential psychometric properties of innovative items. In this study, examinee responses to conventional (e.g., multiple choice) and innovative item formats in a computer‐based testing program were analyzed for IRT information with the three‐parameter and graded response models. The innovative item types considered in this study provided more information across all levels of ability than multiple‐choice items. In addition, accurate timing data captured via computer administration were analyzed to consider the relative efficiency of the multiple choice and innovative item types. As with previous research, multiple‐choice items provide more information per unit time. Implications for balancing policy, psychometric, and pragmatic factors in selecting item formats are also discussed.