THE DEVELOPMENT, INVESTIGATION, AND EVALUATION OF NEW ITEM TYPES FOR THE GRE ANALYTICAL MEASURE

Abstract

ABSTRACTThe aim of this research was to identify, develop, and evaluate empirically new reasoning item types that might be used to broaden the analytical measure of the GRE General Test and to strengthen its construct validity. Item types were identified that varied in the aspects of reasoning they measure. Six item types were selected for empirical evaluation, including the two currently used in the GRE analytical measure. All of the experimental item types were developed in a 3‐option multiple‐choice format, and four of them also were developed in a multiple‐yes/no format. Two experimental batteries were assembled, one using the 3‐option format and the other using the multiple‐yes/no format. Two samples of approximately 370 examinees each, all of whom had recently taken the GRE General Test, were administered one or the other experimental battery. Item analyses and analyses of sex differences, criterion‐related validity, and relationships of the experimental item types to the current GRE measures were conducted. All but one of the experimental item types exhibited promise for strengthening the GRE analytical measure, and even the one exception appeared to be a possible item type for the GRE verbal measure. Evidence for interactions between item type and item format suggested that varying the format may result in the assessment of a different aspect of reasoning for some but not all reasoning item types. Different combinations of the experimental item types were evaluated in a series of confirmatory factor analyses, supplemented by correlational analyses and an exploratory factor analysis. Findings indicated that the convergent validity of the GRE analytical measure probably can be strengthened by selectively adding or substituting some of the experimental item types. However, such alterations of the GRE analytical measure probably would not improve the measure's discriminant validity. The study also provided evidence suggesting that the reasoning domain consists of two major subdomains: informal reasoning and formal‐deductive reasoning. This outcome has implications both for understanding the structure of the reasoning domain and for predicting the impact of different combinations of reasoning item types on the construct validity of the GRE analytical measure.