Graduate Management Admission Test Research Articles

What is the Relevance of the Gmat Analytical Writing Assessment for Management Education?

This commentary is the first of two articles on the Analytical Writing Assessment (AWA), a new component in the Graduate Management Admission Test (GMAT). Here, we examine the AWA's historical significance and address questions regarding the relevance of the new writing test for management education. A comparison of what the AWA measures with GMAC's claims and business schools' expressed needs suggests that the AWA furnishes new information indicating MBA applicant's academic abilities, especially critical thinking and analytical writing abilities. However, as a writing test the AWA predicts students' performance in only one mode of writing required by MBA studies, and it is of little relevance to management writing in the workplace. Overall, we find the AWA to be a limited academic writing assessment that will meet only some of the expectations of management educators.

Analytic Measures for Evaluating Managerial Writing

The recent addition of a writing performance assessment to the Graduate Management Admission Test (GMAT) means that many students now enter business school with a writing assessment score and perhaps even a heightened awareness that writing matters in some way to the successful completion of an MBA degree. This situation presents teachers of business and managerial writing with a new opportunity and pressure to provide students with writing tools that are directly relevant to their business studies and professional careers. The Analysis of Argument Measure and the Persuasive Adaptiveness Measure introduced here are assessment tools that may be used to explain holistic assessment scores (which students receive on the GMAT writing component) and may assist students in understanding and evaluating their writing, both in school and in the workplace. Designed to evaluate managerial documents that are persuasive and directorial in nature, these measures were developed through a series of pilots and used to assess a selected sample of managerial memorandums that were also scored holistically. Correlating the holistic and analytic scores revealed a positive association, and interrater reliability achieved good agreement beyond chance. These results suggest that the measures may be reliably employed to assess characteristics valued in managerial writing. Examples of how these analytic measures may be employed for teaching and research are also described.

DIFFERENTIAL ITEM FUNCTIONING ON THE GRADUATE MANAGEMENT ADMISSION TEST

ABSTRACTThe purpose of this study was to identify differentially functioning items on operational administrations of the Graduate Management Admission Test (GMAT) through the use of the Mantel‐Haenszel statistic. Retrospective analyses of data collected over a three‐year period are reported for Black/White and female/male comparisons. Two tests are included in the analyses–Verbal and Quantitative. Data concerning the number of items identified as differentially difficult, the characteristics of the test questions associated with differential item difficulty, and the relationship of item difficulty to differential item functioning are discussed. In general, one percent to six percent of the items were identified as being differentially difficult per comparison with a greater number of items flagged in the female/male analyses than in the Black/White analyses. There was little overlap among the items flagged in the two analyses. Although the analyses suggested some content characteristics that may be related to differential item functioning, these findings about the GMAT items should be considered tentative since only a small number of items was studied and all investigations were post hoc analyses. Correlations between item difficulty and differential item functioning were generally low, with the exception of Quantitative items in the Black/White analyses. For these items, a moderately positive relationship existed between item difficulty and the differential item functioning statistic, showing that Black examinees performed differentially better than matched White examinees as item difficulty increased.

Predicting Academic Success of Students in a Master of Business Administration Program

The results of a study to correlate graduate school of business entrance criteria with success in a Master of Business Administration program (as measured by graduate GPA) are reported. Two groups of graduate students ( n = 50 for each group) were randomly selected from a pool of 203 recent graduates. One group had taken the Graduate Management Admissions Test (GMAT); the second group, the Miller Analogies Test (MAT). Ten predictor variables were used in bivariate and stepwise multiple regression analyses. The 10 variables were number of semesters in the program, undergraduate GPA, age, ethnic background, sex, marital status, GMAT score, MAT score, number of years since undergraduate degree completion, and the type of undergraduate degree earned (BS vs. BA). Results revealed a strong correlation between the GMAT score and graduate GPA, especially when combined with undergraduate GPA. The relationship between MAT score and graduate GPA was significant, but not as strong as that between GMAT and graduate GPA. There was a strong relationship between ethnic background and graduate GPA when using the MAT.

Segmenting the MBA Market:

This paper discusses segmentation of the potential MBA student market according to demographic criteria, based on the assumption that student success (identified as completion of the degree within five years) is indicative of client satisfaction. The University of Western Australia's MBA program began in 1973. Using the records for this period, demographic variables are tested as predictors of student success. It is found that students who have been admitted without taking the Graduate Management Admissions Test (GMAT) do significantly worse than average, as do those students whose previous degrees were in the professional fields of Medicine, Law and Architecture. Women do less well than men, though this is explained by the facy that a larger percentage of women have been admitted without GMAT scores. It is also found that, for those students admitted to the couse, the GMAT score (except for very high scores) and the years of work experience are not significantly related to success. This result is interpreted as indicating, not that GMAT score and work experience are unrelated to success, but rather that these criteria have been appropriately weighed against other criteria in the selection procedure.

THE CORRECTION FOR GUESSING: A FURTHER ANALYSIS OF ANGOFF AND SCHRADER

This study reexamines results reported by Angoff and Schrader (1981, 1984) regarding formula directions and rights directions for standardized tests. In that study, it was concluded that the two scoring directions were essentially equivalent. In this study, methodological concerns are discussed and additional data analyses undertaken. Among the methodological concerns discussed are the potential problems in using volunteers for the College Board phase of the study and the likelihood of treatment contamination in the Graduate Management Admissions Test (GMAT) phase. Estimates of success rates of the rights directions group on items omitted and not reached by the formula group, were beyond chance levels by 0.6 % to 13.0 % for the College Board tests and at or below chance levels for the GMAT. Alternative interpretations of the data and suggestions for additional research are proposed.

AN EXPLORATORY STUDY OF THE APPLICABILITY OF ITEM RESPONSE THEORY METHODS TO THE GRADUATE MANAGEMENT ADMISSION TEST1

ABSTRACTA necessary prerequisite to the operational use of item response theory (IRT) in any testing program is the investigation of the feasibility of such an approach. This report presents the results of such research for the Graduate Management Admission Test (GMAT).Despite the fact that GMAT data appear to violate a basic assumption of the three‐parameter logistic item response model, local independence, the model was able to replicate accurately the observed item responses. IRT‐based equating was consistent across two randomly selected samples and four selected subpopulations (male, female, younger examinees, and older examinees) and produced converted scores very similar to those produced by the current GMAT equating method — linear section pre‐equating — a method that makes different assumptions than those required by IRT. It appears that for GMAT item types and populations, any effect of the violation of local independence on IRT true‐score equating is negligible.This research has shown IRT equating to be feasible for the GMAT; but, because the local independence assumption of IRT appears to be violated, further experience is needed before other IRT methods — such as optimal test development, using item information, or computerized adaptive testing — could be used for the GMAT.

FACTORS AFFECTING GMAT PREDICTIVE VALIDITY FOR FOREIGN MBA STUDENTS: AN EXPLORATORY STUDY

ABSTRACTThe primary aim of this cooperative study was to assess the role of selected English‐proficiency related test and background variables as moderators of the relationship between scores on the Graduate Management Admission Test (GMAT), and first year average grade (FYA) in samples of foreign MBA students. The study was guided by working hypotheses specifying that GMAT/FYA correlations would be systematically higher for students with higher levels of English proficiency than for those with lower levels of English proficiency. Levels of proficiency were defined operationally (a) by native‐English speaking vs non‐native English speaking status, (b) score levels on the Test of English as a Foreign Language (TOEFL), and (c) score levels on a Relative Verbal Performance Index (RVPI)—a derived test variable reflecting level of GMAT verbal score relative to that expected for U.S. examinees with given quantitative scores. It was also hypothesized that GMAT/FYA correlations would be higher (a) for students from countries whose U.S.‐bound nationals typically earn higher average scores on TOEFL than for students from countries with typically lower‐scoring student contingents, and (b) for samples that were completely homogeneous with respect to country of citizenship than in more general samples.Data were supplied by 59 U.S. schools of management for 1,762 foreign non‐native speakers of English (English second language or ESL) and 157 foreign native speakers (English primary language or EPL). Continuous variables (e.g., GMAT verbal and quantitative score, TOEFL Total score, FYA, and so on) were standardized by school—that is, expressed as deviations from school means in school standard deviation units—and then pooled for analysis.GMAT quantitative scores were found to be more valid than GMAT verbal scores for essentially all subgroups of foreign students. For the EPL foreign sample, GMAT V/FYA and Q/FYA correlations were similar to medians reported by the Graduate Management Admission Council (GMAC) validity study service (VSS) for 85 samples of U.S. MBA students; this was true as well for ESL students scoring 600 or higher on TOEFL and for those in the upper two‐thirds with respect to the RVPI index. GMAT/FYA correlations were higher for students from European countries, and from Asian countries with an established English‐speaking tradition (e.g., India, Malaysia, the Philippines), than for students from other Asian countries (e.g., Taiwan, Japan, Indonesia, Korea) and the Middle East. TOEFL/FYA and GMAT V/FYA correlations were similar.It was found that the mean relative standing of various country‐contingents in terms of first year grades tended to correspond more closely with relative standing on the less valid verbal measure than with standing on the more valid quantitative measure. One conclusion was that for these samples of non‐native speakers of English, differences in English‐language background affected (artifactually depressed) both performance on the GMAT verbal measure and first‐year performance in the MBA programs.Findings suggested that a set of subgroup prediction systems would likely be better than any general system for foreign MBA students.

A STUDY OF HYPOTHESES BASIC TO THE USE OF RIGHTS AND FORMULA SCORES

The hypothesis that some students, when tested under formula directions, omit items about which they have useful partial knowledge implies that such directions are not as fair as rights directions, especially to those students who are less inclined to guess. This hypothesis may be called the differential effects hypothesis. An alternative hypothesis states that examinees would perform no better than chance expectation on items that they would omit under formula directions but would answer under rights directions. This may be called the invariance hypothesis. Experimental data on this question were obtained by conducting special test administrations of College Board SAT‐verbal and Chemistry tests and by including experimental tests in a Graduate Management Admission Test administration. The data provide a basis for evaluating the two hypotheses and for assessing the effects of directions on the reliability and parallelism of scores for sophisticated examinees taking professionally developed tests. Results support the invariance hypothesis rather than the differential effects hypothesis.

ESTIMATING THE RELATIONSHIP BETWEEN USE OF TEST‐PREPARATION METHODS AND SCORES ON THE GRADUATE MANAGEMENT ADMISSION TEST

ABSTRACTThis study sought to examine the relationship between five methods of test preparation and test performance as measured by Graduate Management Admission Test (GMAT) Verbal (V), Quantitative (Q) and Total (T) scores. Data on method of test preparation were obtained through voluntary examinee response to the following five questions which appeared on the answer sheets:In preparing for this test, did you: Study the sample questions in the GMAT registration bulletin? Work through an actual GMAT published by ETS? Use a book not published by ETS on how to prepare for the GMAT? Attend a test preparation or coaching course for the GMAT? Undertake on your own any review of mathematics? One sample of first‐time test takers and one sample of second‐time test takers were selected from among the 185,525 1981–82 GMAT examinees who were U.S. citizens. Multiple regressions using GMAT scores as dependent variables and test preparation, undergraduate grade point average (UGPA) and sex as independent variables were computed separately for first‐time examinees who were members of the Afro‐American/Black, Caucasion/White, Oriental/Asian and Spanish‐American U.S. citizen subgroups. Regressions (including first GMAT scores as independent variables) were also computed for all examinees in the sample who were taking the GMAT for the second time.FIRST‐TIME EXAMINEESThe percents of first‐time examinees electing to use each method varied, but the rank order of the frequency of using each method was consistent across the subgroups. The largest proportion of first‐time examinees reported that they had prepared by reviewing the bulletin, followed in descending order by using a test preparation book not prepared by ETS, undertaking their own study of mathematics, working through an actual GMAT, and attending a test preparation course. The study also found that examinees electing to use the various methods of preparation did not differ appreciably in previous academic performance as measured by undergraduate grade point average, but did vary slightly in age and amount of work experience.Results of the multiple regression analyses based on data from first‐time examinees differed across the four subgroups. The size of the coefficients associated with each method of preparation, as well as the corresponding standard errors, varied among the four subgroups. The same was true for the interaction effects between pairs of methods. The expected difference in verbal score for a “yes” response to “Studying a test review book not published by ETS” when the effects of the other independent variables were held constant, ranged from 1.6 to 3.2 scaled score points and were significant for all four subgroups. The difference in verbal scores for “Studying the Bulletin” ranged from 1.3 scaled score points for Afro‐American/Blacks to 4.0 scaled points for Oriental/Asians. The effects of using a review book or taking a review course ranged between .4 and 1.9 points on verbal and quantitative scores. Negative effects were associated with examinees' own review of mathematics. These effects were attributed to a confounding between self‐selection and method of preparation.SECOND‐TIME EXAMINEESThe sizes of the effects associated with each method of preparation for second‐time examinees were considerably less than those obtained using data from first‐time examinees. When previous test performance was held constant, the effect of using each of the methods of preparation was small. In fact, only the effects of using a test preparation book and of attending a test preparation course were significantly different from zero.The mean GMAT scores between second‐time examinees who did and did not use each method of preparation differed inconsistently and only slightly. Additionally, the magnitude of gain over the first administration score was very similar between examinees who used and did not use each method.CONCLUSIONSThis study has indicated that differences in GMAT scores do exist among examinees using different methods of preparing for the examination. However, it was shown that when initial ability, as measured by GMAT first score, was controlled, the sizes of the effects of studying the GMAT bulletin, working through an actual GMAT, and reviewing mathematics were not significantly different from zero. The effects of the methods on GMAT scores of first‐time examinees, for whom a previous score was not available to use as a covariate, were larger. In those analyses (in which self‐reported UGPA was used as a less effective control on ability), the largest effects associated with any method over the other were about 4, 3 and 33 verbal, quantitative and total score points, respectively. However, the effects of using these methods are confounded with the characteristics of examinees who choose to use each method. The effects resulted from a combination of self‐selection and preparation. There do appear to be relationships between method of preparation and test scores. However, it must be emphasized that it does not necessarily follow that using any of the methods of preparation causes an increase in scores.

GMAT AND GRE APTITUDE TEST PERFORMANCE IN RELATION TO PRIMARY LANGUAGE AND SCORES ON TOEFL

ABSTRACT This study was designed to describe and analyze (1) the, performance of foreign candidates taking the Graduate Management Admission Test (GMAT) or the Graduate Record Examinations (GRE), Aptitude Test in relation to self-reported 'primary language (English vs.,.other), and (2) relationships between performance on the .respective.admissions teats and performance on the Test of English as eforeign Languages (TOEFL) for subgroups of foreign admissions-test Candidates identified by Cross-file matching es having. also taken .TOEFL. Data were obtained from files maintained by the three testing programs and analyses were based on data for examinees tested during the period from September 1977N:through August .1979. (Authdr/PN)

Ability Testing and Graduate Admissions: Decision Process Modeling and Validation

The admissions process for a graduate Master of Business Administration (MBA) program was modeled for 406 applicants to a newly designed MBA program. The decision process model was then applied to a sample of students admitted to and enrolled in an old MBA program, where less stringent admissions procedures had been used. Results supported the adequacy of both Graduate Management Admission Test scores and undergraduate grades in the prediction of admissions officer decisions as well as the validity of such information in the prediction of graduate performance and turnover. The results are discussed in light of the recent criticisms of the use of standardized tests as admissions criteria for programs of higher education.

UNBIASED PREDICTIONS IN SPARSE DATA PROBLEMS

ABSTRACTThis paper presents a brief synopsis of some recent work of the authors (Braun and Jones, 1981) in connection with the use of regression methods to predict performance in graduate schools of management on the basis of undergraduate grade point average (UGPA) and the Graduate Management Admission Test (GMAT) score. The central problem is how to provide stable unbiased predictions for identifiable groups of candidates whose members may be sparsely distributed across the institutions under study. It is shown how empirical Bayes methods provide a workable solution even when classical procedures can not be implemented.

GMAT Scores and Performance: Selecting Students into a Graduate Management School

Research shows that scores on the Graduate Management Admission Test (GMAT) are positively correlated with academic performance at graduate management schools. This study replicates this finding on Australian data and extends the analysis to cover performance at the individual course level as well as for aggregate grade point average. This traditional treatment of the data while directed at the admission decision is in fact structured to explain variance in performance. A novel analysis is presented in parallel which investigates the trade–off between Type I errors (rejecting a student who would pass) and Type II errors (accepting a student who fails). This reveals that, while GMAT may be a good predictor of performance, it is an inefficient discriminator for selection purposes.

The Utility of Discriminant Analysis for Predicting Graduation From a Master of Business Administration Program

With universities facing enrollment ceilings and with increased competition for admission to graduate programs, it has become increasingly important for admissions committees to select those applicants who are most likely to complete their degrees. This study was concerned with the accuracy with which successful completion of the Master of Business Administration (MBA) degree could be predicted from readily available admissions data: sex of the student, student age, undergraduate grade point average, Graduate Management Admission Test (GMAT) Verbal score, and GMAT Quantitative score. As a mechanism for evaluating the predictive accuracy of the MBA admissions data, discriminant analysis was used. The sample for this study consisted of all students who had begun and in some way had concluded their MBA studies between 1969 and 1979. This total sample ( N = 507) was randomly split into two parts, one part for deriving the discriminant function and one part for cross-validating the derived discriminant function. The results of the dis criminant analysis clearly showed that the two criterion groups (graduates and non-graduates) could be differentiated. In addition, the utility of the discriminant function derived from the one sample was demonstrated on the cross-validation sample. Of the students in this hold-out sample, 69% of those predicted to graduate actually did.

A STUDY OF ALTERNATIVE METHODS FOR EQUATING RIGHTS SCORES TO FORMULA SCORES

The principal purpose of this study was to determine whether it would be possible to eqtate Rights-scored tests to Formula-scored tests without causing a discontinuity in the meaning of the score scale. Several other subsidiary studies -of the characteristics of the two scoring methods, of nonresponse and guessing, and of reliability and pardilelism--were also indertaken. The study was conducted in two phases: 1) of two forms of the SAT-verbal and one form of the College Board Chemistry Achievement Test, based on data from a specia:t experimental test administration; and 2) of operational and experimental subtests of the Graduate Management Admission Test, based on data from the (regular) October 1980 administration of the GMAT to applicants for business school. Several outzomes of the study were observed that will be useful for the understanding of some of the issues of Rights and Formula scoring. In addition, it was found that the data of this study support the hypothesis that Formula scores for tests administered with Rig!its directions are directly comparable to Formula scores for the fme tests administered with Formula directions. Thus, the directions under which a test is administered can be,changed without serious concern that a discontinuity in the score scale will result.

CONSTRUCT VALIDITY OF THE GMAT: A FACTOR ANALYTIC STUDY*

ABSTRACTA factor analytic study of the Graduate Management Admission Test was undertaken in order to gain a more thorough understanding of the psychological constructs underlying examinee performance on the test. The degree to which various dimensions or explanatory concepts account for test performance was established for three alternate forms of the GMAT and for several subgroups of examinees in order to determine the degree of equivalence of test forms and the similarity of measured constructs for various groups. Dimensions of test variation emerged that were thought to be useful to test development staff in sharpening test content specifications needed for the development of future equivalent forms of the test. One dimension that emerged which currently is not explicit in the GMAT specifications was the distinction between quantitative items that are abstract in nature and those that have a practical or applied nature. The relationship of each of three experi mental item types to the factors underlying the current operational test was also determined through factor extension analysis and recommendations were made regarding the use of these three new item types in future revisions of the GMAT.

THE RELATIONSHIP BETWEEN SCORES ON THE GRADUATE MANAGEMENT ADMISSION TEST AND THE TEST OF ENGLISH AS A FOREIGN LANGUAGE

ABSTRACTIn addition to scores on the Graduate Management Admission Test (GMAT), which are required of applicants to a substantial number of graduate management schools, foreign candidates may also be required to submit scores on the Test of English as a Foreign Language (TOEFL) as an Indication of English language proficiency.The present study provides an analysis of the relationships among TOEFL and GMAT total scores and subscores for a large sample of foreign GMAT candidates. The results suggest that the apparent “discrepancies” between TOEFL scores and GMAT verbal scores are, because of differences in score scales, norming populations, intended purposes, and levels of difficulty, to be expected. The relationships (and “discrepancies”) between TOEFL and GMAT scores are relatively similar for candidates from several countries.

DIFFERENCES IN ITEM PERFORMANCE ACROSS GROUPS

ABSTRACTAn analysis of the Graduate Management Admission Test was undertaken to isolate factors that make some items unexpectedly easy or difficult for minority candidates relative to the majority candidate, a young male Caucasian of U. S. citizenship. Minority groups were selected within the following dimensions: sex, ethnic background or race, age, and language fluency. Fourteen pairwise group comparisons were completed and all sections of a current test form analyzed. The group comparisons were of item difficulty. An iterative technique was adopted to determine a line of best fit for plots of difficulty for one group against difficulty for another. Empirical criteria were established for identifying outlying items as unexpectedly easy or difficult. In addition to describing the study and its results, the research report includes a discussion of how the methodology may be extended to the simultaneous comparison of any number of groups and an analysis of the sampling error in calculations of item difficulty.