Test Development Practices Research Articles

When the Test Developer Does Not Speak the Target Language: The Use of Language Informants in the Test Development Process

ABSTRACTIt is not unusual for tests in less-commonly taught languages (LCTLs) to be developed by an experienced item writer with no proficiency in the language being tested, in collaboration with a language informant who is a speaker of the target language, but lacks language assessment expertise. How this approach to item writing works in practice, and what factors play a role in it, is largely unrecorded, as are item-writing processes and practices in language assessment in general. Through a case study approach, this study sought to gain insights into test development practices in cases when essential item writer traits are spread across different people. Seven in-depth interviews with language assessment specialists and language informants involved in LCTL reading test development revealed a number of specific characteristics, and also challenges, to test developer recruitment and test development in this context. Findings indicate that this inherently collaborative approach brings with it a sophisticated system of “checks and balances” that may benefit item writing in some respects.

Differential Performance on National Exams

The purpose of this study is to evaluate two methodological perspectives of test fairness using a national Secondary School Certificate (SSC) examinations. SSC is a suit of multi-subject national qualification tests at Grade 10 level in South Asian countries, such as Bangladesh, India, and Pakistan. Because it is a high-stakes test, the fairness of SSC tests is a major concern among public and educational policy planners. This study is a first attempt to investigate test fairness of the national SSC examination of Pakistan using two independent differential item functioning (DIF) and differential bundle functioning (DBF) procedures. The SSC was evaluated for possible gender bias using multiple-choice tests in three core subjects, namely, English, Mathematics, and Physics. The study was conducted in two phases using explanatory item response model (EIRM) and Simultaneous Item Bias Test (SIBTEST). In Phase 1, test items were studied for DIF, and items with severe DIF were flagged in each subject. In Phase 2, the item bundles were analyzed for DBF. Three items were detected with large DIF, one for each subject, and one item bundle was detected with a negligible DBF. Taken together, the results demonstrate that there is no major threat to the validity of the interpretation of examinees’ test scores on the SSC examination. The outcome from this study provided evidence for test fairness, which will enhance test development practices at the national examination authorities.

An introduction to the use of evidence-centered design in test development

The purpose of this article is to describe what Evidence-Centered Design (ECD) is and to explain why and how ECD is used in the design and development of tests. The article will be most useful for readers who have some knowledge of traditional test development practices, but who are unfamiliar with ECD. The article begins with descriptions of the major characteristics of ECD, adds a brief note on the origins of ECD, and discusses the relationship of ECD to traditional test development. Next, the article lists the important advantages of using ECD with an emphasis on the validity of the inferences made about test takers on the basis of their scores. The article explains the nature and purpose of the “layers” or stages of the ECD test design and development process: 1) domain analysis; 2) domain modeling; 3) conceptual assessment framework; 4) assessment implementation; and 5) assessment delivery. Some observations about my experience with the early application of ECD for those who plan to begin using ECD, a brief conclusion, and some recommendations for further reading end the article.

Voices from stakeholders: What makes an academic English test ‘international’?

Recent years have witnessed a growing number of speakers of all varieties of English in English-medium higher education. It has also become increasingly common for an academic English test to be used and accepted in multiple countries for admissions purposes. Both trends have implications for the definition of the target language use domain for tests of English for academic purposes, and have brought challenges to traditional test development practices based on one of the Standard varieties of English. In this study we investigated what properties a global academic English test should include from the perspectives of multiple stakeholder groups, including test takers, English teachers, score users, and language testing professionals. Survey responses from the participants indicated both willingness to embrace diversity and reservations about accommodating differences. Further analysis by stakeholder indicated that perceptions varied to a certain extent across stakeholder groups. The results also indicated that various factors were recognized as contributing to an academic test's global perception. The findings contribute to our understanding of assessing academic English in the global context. The results can also inform the alignment of testing practices with the reality of English language use globally, which suggests implications for test development and delivery.

An Introspective Study of Arabic and Mandarin-Speakers’ Reading Comprehension Strategies

Little L2 reading strategy research has explored the effect of linguistic and cross-cultural differences on strategic reading habits. This study attempted to fill this void by examining the reading strategies that Arabic and Mandarin speaking immigrants employed when reading and answering Canadian Language Benchmarks Assessment reading comprehension items. A chi-square analysis of their bottom-up and top-down strategy use revealed that the Mandarin-speakers used bottom-up strategies significantly more than expected, whereas the Arabicspeakers used top-down strategies more than expected. The findings of the study are discussed in the light of earlier research, and some preliminary implications for reading strategy instruction, strategy training studies, and test development practices are suggested.

Simulation-based Assessment in Anesthesiology

Simulations have taken a central role in the education and assessment of medical students, residents, and practicing physicians. The introduction of simulation-based assessments in anesthesiology, especially those used to establish various competencies, has demanded fairly rigorous studies concerning the psychometric properties of the scores. Most important, major efforts have been directed at identifying, and addressing, potential threats to the validity of simulation-based assessment scores. As a result, organizations that wish to incorporate simulation-based assessments into their evaluation practices can access information regarding effective test development practices, the selection of appropriate metrics, the minimization of measurement errors, and test score validation processes. The purpose of this article is to provide a broad overview of the use of simulation for measuring physician skills and competencies. For simulations used in anesthesiology, studies that describe advances in scenario development, the development of scoring rubrics, and the validation of assessment results are synthesized. Based on the summary of relevant research, psychometric requirements for practical implementation of simulation-based assessments in anesthesiology are forwarded. As technology expands, and simulation-based education and evaluation takes on a larger role in patient safety initiatives, the groundbreaking work conducted to date can serve as a model for those individuals and organizations that are responsible for developing, scoring, or validating simulation-based education and assessment programs in anesthesiology.

The cultural malleability of intelligence and its impact on the racial/ethnic hierarchy.

This commentary highlights previous literature focusing on cultural and environmental explanations for the racial/ethnic group hierarchy of intelligence. Assumptions underlying definitions of intelligence, heritability/genetics, culture, and race are noted. Historical, contextual, and testing issues are clarified. Specific attention is given to studies supporting stereotype threat, effects of mediated learning experiences, and relative functionalism. Current test development practices are critiqued with respect to methods of validation and item development. Implications of the genetic vs. culture-only arguments are discussed with respect to the malleability of IQ. Rushton and Jensen (2005) review decades of literature to support a genetic basis for the racial/ethnic group hierarchy in intelligence, a position they have held unwaveringly for over 30 years. Their report gives little mention to findings that point to the impact of environment and race (i.e., race as a social construction) on intellectual development or performance—what they term the culture-only perspective. We are not among the culture-only adherents as characterized by Rushton and Jensen. While acknowledging the impact of biological factors on intelligence test performance, we have examined the impact of cultural/environmental factors that affect performance on aptitude and achievement measures. Our work, and that of others (e.g., Aronson, 2002; Sternberg, 1996), show us that intellectual performance is much more fragile and malleable than what is often noted in the current literature. The goals of our commentary are to highlight, briefly, assumptions underlying definitions (i.e., intelligence, heritability, genetics, culture, race) and clarify historical, contextual, and testing issues that were only briefly mentioned by Rushton and Jensen. Finally, we comment on the heuristic value and on policy implications of the research.

Literacy and Literacies: Texts, Power, and IdentityJames Collins and Richard K. Blot, Cambridge, UK: Cambridge University Press, 2003, 217 pp., ISBN: 0-521-59661-0

Recent years have witnessed a growing number of speakers of all varieties of English in English-medium higher education. It has also become increasingly common for an academic English test to be used and accepted in multiple countries for admissions purposes. Both trends have implications for the definition of the target language use domain for tests of English for academic purposes, and have brought challenges to traditional test development practices based on one of the Standard varieties of English. In this study we investigated what properties a global academic English test should include from the perspectives of multiple stakeholder groups, including test takers, English teachers, score users, and language testing professionals. Survey responses from the participants indicated both willingness to embrace diversity and reservations about accommodating differences. Further analysis by stakeholder indicated that perceptions varied to a certain extent across stakeholder groups. The results also indicated that various factors were recognized as contributing to an academic test's global perception. The findings contribute to our understanding of assessing academic English in the global context. The results can also inform the alignment of testing practices with the reality of English language use globally, which suggests implications for test development and delivery.

The Lack of Fidelity Between Cognitively Complex Constructs and Conventional Test Development Practice

Do current test development practices align well with the cognitively complex constructs being called for in the educational reform movement? What types of test development practices are needed to develop measures of cognitively complex constructs?

AN INVESTIGATION OF THE ROLE OF EDUCATIONAL BACKGROUND ON PERFORMANCE ON THE GRE® ANALYTICAL MEASURE

ABSTRACTThere have been a number of reports that educational background is related to performance on reasoning problems similar to those used on the GRE analytical measure. The purpose of this research was to evaluate whether educational background as defined by broad undergraduate major fields was related to differential difficulty for specific items on the GRE analytical measure. An index of differential difficulty, MH‐DDIF, was calculated for the reasoning items from 13 GRE forms. We found only low to moderate levels of MH‐DDIF for the two item types on the analytical measure. A higher proportion of logical reasoning items than analytical reasoning items had moderate MH‐DDIF. The relationship of item classification attributes to MH‐DDIF was evaluated. In addition, other attributes potentially related to MH‐DDIF for logical reasoning items were identified and their ability to predict MH‐DDIF was evaluated with a cross‐validation design. The attributes identified as potentially related to MH‐DDIF for items from one subset of GRE forms did not predict MH‐DDIF for items in a second subset. Thus, a strong link between item performance and educational background was not observed within item types. Because of the failure to cross‐validate the potential predictors of DIF, other factors that contribute to MH‐DDIF are briefly considered. The relationship of these results to other research on the effect of educational background is discussed. We also discuss implications for test design and how test development practices are effective in minimizing the impact of educational background within item types.

EFFECTS OF PROSE COMPLEXITY ON ACHIEVEMENT TEST ITEM DIFFICULTY

Better understanding of sources of difficulty in test items would improve the test development process by bringing the functioning of items more under the control of the test developer. To help increase this understanding, a study was undertaken to evaluate the effects of various aspects of prose complexity on the difficulty of achievement test items. The items of interest were those that presented a verbal stimulus followed by a question about the stimulus and a standard set of multiple‐choice options. Items were selected for study from two tests with differing demands on an examinee's knowledge base, NTE Communications Skills and GRE Subject Test in Psychology. Standard multiple regression analyses and Embretson's model fitting procedures were used to evaluate the contribution of various complexity factors to the prediction of difficulty. These factors, which included measures of item structure, readability, semantic content, cognitive demand and knowledge demand, were found to be successful in predicting item difficulty for these items. The immediate usefulness of the results for test development practice, however, are limited by the fact that only a single item type was studied and by the time required to develop the complexity measures.

The Relationship of Content Characteristics of GRE Analytical Reasoning Items to Their Difficulties and Discriminations

In actual test development practice, the number o f test items that must be developed and pretested is typically greater, and sometimes much greater, than the number that is eventually judged suitable for use in operational test forms. This has proven to be especially true for one item type–analytical reasoning‐that currently forms the bulk of the analytical ability measure of the GRE General Test. This study involved coding the content characteristics of some 1,400 GRE analytical reasoning items. These characteristics were correlated with indices of item difficulty and discrimination. Several item characteristics were predictive of the difficulty of analytical reasoning items. Generally, these same variables also predicted item discrimination, but to a lesser degree. The results suggest several content characteristics that could be considered in extending the current specifications for analytical reasoning items. The use of these item features may also contribute to greater efficiency in developing such items. Finally, the influence of these various characteristics also provides a better understanding of the construct validity of the analytical reasoning item type.

CONTENT CHARACTERISTICS OF GRE ANALYTICAL REASONING ITEMS

ABSTRACTIn actual test development practice, the number of test items that must be developed and pretested is typically greater, and sometimes much greater, than the number that is eventually judged suitable for use in operational test forms. This has proven to be especially true for one item type–analytical reasoning–that currently forms the bulk of the analytical ability measure of the GRE General Test.This study involved coding the content characteristics of some 1,400 GRE analytical reasoning items. These characteristics were correlated with indices of item difficulty, discrimination, and independence from the verbal and quantitative measures.Several item characteristics were predictive of the difficulty of analytical reasoning items. Generally, these same variables also predicted item discrimination, but to a lesser degree. Independence from the GRE verbal and quantitative measures was largely unpredictable.The results suggest several content characteristics that could be considered in extending the current specifications for analytical reasoning items. The use of these item features may also contribute to greater efficiency in developing such items. Finally, the influence of these various characteristics also provides a better understanding of the construct validity of the analytical reasoning item type.