Validity Study Service Research Articles

The Validity of Graduate Management Admission Test Scores

The validity of Graduate Management Admission Test (GMAT) scores is examined by summarizing 273 studies conducted between 1997 and 2004. Each of the studies was conducted through the Validity Study Service of the test sponsor and contained identical variables and statistical methods. Validity coefficients from each of the studies were corrected for restriction of range on the predictors. The interquartile range of the validity coefficients for the combination of GMAT scores and undergraduate grades in predicting early graduate school performance was .45 to .63. Further examination of study findings showed that higher validity coefficients were observed for executive MBA programs compared to either full-time or part-time programs. There appeared to be no measurable differences in results for public versus private institutions. For programs not located at U.S. institutions, quantitative scores showed lower average validity coefficients compared to programs located in the United States.

How Colleges Can Correctly Determine Selection Benefits from the SAT

Since its inception in 1927, the Scholastic Aptitude Test (SAT) has been used by increasing numbers of colleges in the admissions process. In this article, James Crouse and Dale Trusheim argue that the selection benefits colleges derive from using the SAT in their admissions decisions are minimal. They support this argument through a discussion of the prediction equation, supplemental table of predicted grades, and expectancy table currently used by the College Board's Validity Study Service (VSS) to assess the selection benefits of the SAT for individual colleges. Crouse and Trusheim present an analysis of the added information gain from SAT scores, as well as an alternative method for colleges to assess their selection benefits from the SAT. The authors illustrate their method by using Dartmouth College as a case study. Crouse and Trusheim then call for the inclusion of two additional tables in the VSS presentation to college administrators: the Crosstabulation of Predicted Grades Table and the College Outcomes Table. The Crosstabulation of Predicted Grades Table would allow college administrators to identify the level of redundancy in predicted freshman admissions for their particular institution based on high school GPA alone, compared with those based on high school GPA plus SAT scores. The College Outcomes Table would allow college administrators to estimate the impact their use of the SAT actually has on predicting several different student performance outcomes, such as average grades of admitted students, the percentage of admitted students with freshman GPAs above 2.5, and the percentage of admitted students who graduate.

NEGATIVE COEFFICIENTS IN THE GRE VALIDITY STUDY SERVICE

ABSTRACTOperational procedures for the Graduate Record Examinations Validity Study Service (GREVSS) are reviewed, with the emphasis on the problem of frequent occurrence of negative coefficients in the fitted within‐department regressions obtained by the empirical Bayes method of Braun and Jones (1985). Several alterations of the operational procedures are proposed that would reduce the frequency of negative coefficients, and, if desired, completely eliminate them. It is argued, however, that there are no a priori reasons for assuming that all the coefficients are nonnegative. Reports of the fitted within‐department regressions should be based on a single model, that would be found by model exploration. The estimation procedures could be improved by employing more flexible software for modelling between‐department variation.

ANALYSES OF THE PREDICTIVE VALIDITY OF THE SAT® AND HIGH SCHOOL GRADES FROM 1976 TO 1985

ABSTRACTSince 1964, colleges have been sending data to the College Board Validity Study Service in order to determine the degree to which measures used in admissions predict college performance. These studies have allowed for the monitoring of general trends in the relationship of SAT scores and high school grades with freshman grades. Beginning in the early‐1970s and continuing to the mid‐1970s, the observed average correlations of SAT scores with freshman grades increased dramatically. In the decade that followed, the strength of this relationship gradually declined. In order to determine to what extent the observed variations in the average correlations might be associated with the composition of colleges conducting validity studies each year, validity data were analyzed for the enrolling classes of 1976 to 1985. Estimates of change, based on a repeated measures estimation approach utilizing data from colleges conducting more than one predictive validity study within the 10‐year period, and correcting for the restriction of range in SAT scores and high school record, indicated a general linear trend from 1976 to 1985. The correlations of SAT scores with first‐year college grades declined from about .51 to about .47, for a change of approximately .04, There was less change during this period for private colleges, small colleges, and more selective colleges.

THE PREDICTIVE VALIDITY OF THE RESTRUCTURED GRE WITH PARTICULAR ATTENTION TO OLDER STUDENTS

ABSTRACTData from 99 graduate departments that had participated in the Graduate Record Examinations (GRE) Validity Study Service from 1983 through 1985 were analysed with an empirical Bayes regression model that allows each department's mean scores to influence that department's regression coefficients. To be included in the analysis, students had to have taken the GRE General Test on or after October 1981, when the analytical measure was revised.The empirical Bayes procedure yielded validities at least as high as are commonly reported for least squares analyses. More importantly, it yielded meaningful residuals from a single general model fit to all departments. These comparable residuals made it possible to examine deviations for relatively small subgroups, such as male or female humanities students aged 25 and over.The major finding was a significant underprediction of first‐year grade average for older females in all graduate fields. Although it had been predicted that they would do less well than younger students and about as well as males, they in fact earned considerably higher grades than all other groups.It is suggested that graduate admissions committees broaden efforts to identify nontraditional evidence of talent, motivation, and accomplishment in applicants from this group. The GRE data analysis shows that if a larger proportion of returning females were recruited and admitted, they would perform as well or better than currently enrolled younger students.

GENERALIZATION OF GRE GENERAL TEST VALIDITY ACROSS DEPARTMENTS

ABSTRACTThis study of the validity of the GRE General Test used data from predictive validity studies that were conducted by the GRE Validity Study Service (VSS) in 79 graduate departments. The performance criterion was first‐year grades in graduate school. Observed validities were computed, and for each graduate department validities were also estimated for groups at two other stages of selection–applicants for admission to the department, and all GRE takers.Two validity generalization hypotheses were tested. One was that the General Test's validities were equal across studies; the other was that the General Test's validities had equal ratios across studies, that is, that the level of the validities might vary from institution to institution but the ratios would be constant. These hypotheses were applied for VSS groups, applicant groups, and all GRE takers, and implied validities (validities that would be observed if the hypotheses were true) were calculated. When the implied validities were compared to the observed validities, it was found that the assumption of equal validity did not account well for differences in the level of observed validity of the GRE General Test. The equal ratio hypothesis accounted for the observed validities rather well, possibly due to overcapitalization on chance, but departmental discipline was not significantly related to the degree of fit of observed to implied validities.At all levels of selection, the study yielded applicant validities that were predominantly positive. This lends support to the presumption that the General Test's validity is transportable, i.e., institutions that do not use the General Test can, if they adopt it, expect it to prove valid. In view of the scarcity of very low or negative validities, studies revealing such validities should be questioned.

THE RELATIONSHIP OF GRE GENERAL TEST SCORES TO FIRST‐YEAR GRADES FOR FOREIGN GRADUATE STUDENTS: REPORT OF A COOPERATIVE STUDY

ABSTRACTThe present study was designed as a GRE population validity study for foreign students enrolled in U.S. graduate schools. Primary interest was in validity for students for whom English is a second language (foreign ESL students). The objectives of the study were to obtain information regarding the typical within‐department relationship of scores on the GRE General Test to first‐year average grades (FYA) (a) in samples of foreign ESL students that are heterogeneous with respect to linguistic‐cultural‐educational background, (b) in subgroups that are homogeneous with respect to country of origin and associated background variables, and (c) in subgroups classified according to relative level of English proficiency, as measured by scores on the Test of English as a Foreign Language (TOEFL), GRE verbal and analytical performance relative to quantitative performance, and self‐reported English language communication status.The study was based on data for a total of 1,353 foreign ESL students and 42 foreign ENL (English‐native‐language) students from 97 graduate departments in 23 graduate schools. Eighty‐six departments were primarily quantitative–either engineering, math/science, or economics; six were bioscience departments; and five were primarily verbal–four education and one political science. More than 90 different countries were represented in the sample. The majority of students were from Asia. The three largest national contingents (students from India, Taiwan, and Korea) accounted for about one half of all students. Students were highly selected both in terms of quantitative ability and in terms of English proficiency as measured by TOEFL.The department‐level samples were small (median N = 12). In order to obtain reliable estimates of within‐department GRE/FYA relationships, data for similar departments were pooled. GRE and FYA variables were z‐scaled by department before pooling–that is, scores were expressed as deviations from deparment‐level means in department standard deviation units. Primary emphasis was given to analyses based on pooled data for the primarily quantitative departments, because representation of departments from the other fields was limited.Average levels and patterns of GRE/FYA correlations based on pooled data for foreign ESL students from the 86 quantitative departments and the five “verbal” departments were found to be comparable to levels and patterns of coefficients that have been reported by the GRE Validity Study Service (VSS) for a sample composed primarily of U.S. citizens; results for the small sample of bioscience departments were anomalous, due probably to sampling effects. In quantitative fields, quantitative and analytical scores were the strongest predictors; in the verbal fields, verbal scores were strongest. GRE verbal and TOEFL total scores had parallel patterns of validity.Study findings suggested that inferences based on GRE scores regarding the subsequent academic performance of applicants, especially those applying for admission to primarily quantitative departments, are likely to be as valid for foreign ESL applicants as for U.S. applicants. Questions regarding the comparative academic performance of U.S. and foreign students with comparable GRE scores, not addressed directly in this study, call for further research.

GENERALIZATION OF SAT® VALIDITY ACROSS COLLEGES

ABSTRACTThis study, which focused on the validity of the SAT‐V and SAT‐M, used data from 99 validity studies that were conducted by the Validity Study Service of the College Board. In addition to test validities based on first‐year college averages, which were calculated using institutional data, validities for each college were also estimated for two other groups—applicants for admission to the colleges, and all SAT takers. These last two estimates were based on range restriction theory.Substantial validity generalization was found: the assumption that applicant pool validities were all equal, together with sampling variance and the effects of selection, accounted for 36 percent and 34 percent of the variation of the SAT‐V and SAT‐M validities, respectively. The hypothesis of equal validity in pools like those of all SAT takers, plus sampling variance and the effects of selection, accounted for 53 percent and 33 percent of the variation of the SAT‐verbal and SAT‐mathematical validities, respectively. However, significant institutional uniqueness remains, though part of that uniqueness consists of variation in the reliability of first‐year college average.For these data, substantial validity was the rule. The average validities were quite high, rising to .55 for either SAT‐V or SAT‐M true scores for all SAT takers, and 95 percent of the observed validities were above .13 for SAT‐V and .10 for SAT‐M. Values below these may be owing to accidents of sampling, computing errors, or criterion defects, and it should be noted that 95 percent is a conservative standard. Studies with slightly higher validities may be questioned as well, perhaps repeated, and the criterion examined carefully.A hypothesis that validities for SAT‐V and SAT‐M might differ across institutions but have the same ratio was also tested. It was thought that departures from this second assumption might lead to the detection of institutional types. However, significant departures from the equal‐ratio hypothesis tested at the 5 percent level occurred for about 5 percent of the institutions, so no detection of institutional types occurred.

THE VALIDITY OF VARIOUS METHODS OF TREATING MULTIPLE SAT® SCORES

ABSTRACTA review of the literature concerned with validity data and policies for various methods of treating multiple scores is reported, as are analyses of data from the College Board Validity Study Service. The analyses evaluated the use of SAT‐V alone, SAT‐M alone, and the use of both in combination. The methods for treating multiple scores that were studied were to use the score from the administration with the highest V + M, the highest score, the most recent score, and the average. Best results, in terms of the highest average validity, were achieved using average V + M. Also evaluated were weighted combinations where all scores were used, but the highest or most recent received a unique, empirically determined weight. The use of the empirically determined weights was not superior in cross‐validation.The combinations of variables, and the four methods of treating multiple scores that did not involve the determination of empirical weights, were evaluated in regression equations developed using one‐time testers. All treatments of multiple scores resulted in underprediction of actual grades, with the highest score providing the least amount of underprediction. However, the discrepancy between predicted and actual grades varied greatly across institutions.Data from the Student Descriptive Questionnaire (SDQ) were cross tabulated with the number of retests. It was found that first test score‐level, income, and race were all related to the frequency of retesting.

FACTORS AFFECTING GMAT PREDICTIVE VALIDITY FOR FOREIGN MBA STUDENTS: AN EXPLORATORY STUDY

ABSTRACTThe primary aim of this cooperative study was to assess the role of selected English‐proficiency related test and background variables as moderators of the relationship between scores on the Graduate Management Admission Test (GMAT), and first year average grade (FYA) in samples of foreign MBA students. The study was guided by working hypotheses specifying that GMAT/FYA correlations would be systematically higher for students with higher levels of English proficiency than for those with lower levels of English proficiency. Levels of proficiency were defined operationally (a) by native‐English speaking vs non‐native English speaking status, (b) score levels on the Test of English as a Foreign Language (TOEFL), and (c) score levels on a Relative Verbal Performance Index (RVPI)—a derived test variable reflecting level of GMAT verbal score relative to that expected for U.S. examinees with given quantitative scores. It was also hypothesized that GMAT/FYA correlations would be higher (a) for students from countries whose U.S.‐bound nationals typically earn higher average scores on TOEFL than for students from countries with typically lower‐scoring student contingents, and (b) for samples that were completely homogeneous with respect to country of citizenship than in more general samples.Data were supplied by 59 U.S. schools of management for 1,762 foreign non‐native speakers of English (English second language or ESL) and 157 foreign native speakers (English primary language or EPL). Continuous variables (e.g., GMAT verbal and quantitative score, TOEFL Total score, FYA, and so on) were standardized by school—that is, expressed as deviations from school means in school standard deviation units—and then pooled for analysis.GMAT quantitative scores were found to be more valid than GMAT verbal scores for essentially all subgroups of foreign students. For the EPL foreign sample, GMAT V/FYA and Q/FYA correlations were similar to medians reported by the Graduate Management Admission Council (GMAC) validity study service (VSS) for 85 samples of U.S. MBA students; this was true as well for ESL students scoring 600 or higher on TOEFL and for those in the upper two‐thirds with respect to the RVPI index. GMAT/FYA correlations were higher for students from European countries, and from Asian countries with an established English‐speaking tradition (e.g., India, Malaysia, the Philippines), than for students from other Asian countries (e.g., Taiwan, Japan, Indonesia, Korea) and the Middle East. TOEFL/FYA and GMAT V/FYA correlations were similar.It was found that the mean relative standing of various country‐contingents in terms of first year grades tended to correspond more closely with relative standing on the less valid verbal measure than with standing on the more valid quantitative measure. One conclusion was that for these samples of non‐native speakers of English, differences in English‐language background affected (artifactually depressed) both performance on the GMAT verbal measure and first‐year performance in the MBA programs.Findings suggested that a set of subgroup prediction systems would likely be better than any general system for foreign MBA students.

USE OF EMPIRICAL BAYES METHODS IN THE STUDY OF THE VALIDITY OF ACADEMIC PREDICTORS OF GRADUATE SCHOOL PERFORMANCE

ABSTRACTGraduate education in the United States is characterized by an enormous diversity of disciplines and the predominance of relatively small enrollments in individual departments. In this setting, a validity study based on a single department's data and employing classical statistical methods can only be of limited utility and applicability. In order to participate in the Graduate Record Examinations Validity Study Service, a department must have at least 25 students in its entering class. Only validities for single predictors are provided; estimates of the validity of two or more predictors, used jointly, are considered too unreliable because the corresponding prediction equations often possess implausible characteristic, such as negative coefficients. These constraints were introduced by the Validity Study Service to reduce the chance that the results in the report to a department would be overly influenced by statistical artifacts in the data and hence serve more to mislead than to inform.In this study we investigated two statistical methods, empirical Bayes and cluster analysis, to determine whether their application to the problems faced by the Validity Study Service could result in useful improvements. Considerable effort was expended in developing and studying a new and more general class of empirical Bayes models that can accommodate the complex structure of the Validity Study Service data base.The principal methodological conclusions of this study are that, through the use of a new class of empirical Bayes methods, it is possible to obtain, at the departmental level, useful and reliable estimates of the joint validity of several predictors of academic performance and that these methods may be further refined to address the question of differential predictive validity, again at the departmental level. These results have important practical implications for the GRE Validity Study Service.

Express Preparation for the SAT

"Express Preparation for the SAT." The Clearing House: A Journal of Educational Strategies, Issues and Ideas, 57(6), pp. 269–270 NotesA sound statistical study is R. A. Weitzman’s “The Prediction of College Achievement by the Scholastic Aptitude Test and the High School Record,” Journal of Educational Measurement 19 (Fall 1982). Both the College Board and the Educational Testing Service (ETS) have also published convincing, if less disinterested, studies of the correlation between SAT scores and college achievement. See, for example, S. F. Ford and S. Compos, Summary of Validity Data from the Admissions Testing Program Validity Study Service (Princeton: College Entrance Examination Board, 1977).

Grade Inflation and the Validity of the Scholastic Aptitude Test

Grade inflation refers to an increase in grade point average without a concomitant increase in achievement. Therefore, to show that grade inflation exists it is important to demonstrate that achievement has not increased while grades have increased. One purpose of this study is to clarify this issue by examining the data base of the College Board's Validity Study Service. The second purpose of the study is to examine the effect of grade inflation on the validity of the Scholastic Aptitude Test across a period of 15 years.

Grade Inflation and the Validity of the Scholastic Aptitude Test

Grade inflation refers to an increase in grade point average without a concomitant increase in achievement. Therefore, to show that grade inflation exists it is important to demonstrate that achievement has not increased while grades have increased. One purpose of this study is to clarify this issue by examining the data base of the College Board's Validity Study Service. The second purpose of the study is to examine the effect of grade inflation on the validity of the Scholastic Aptitude Test across a period of 15 years.

A SUMMARY OF THE RESULTS OF THE GRADUATE MANAGEMENT ADMISSION COUNCIL (GMAC) VALIDITY STUDY SERVICE FOR 1979–1980

ABSTRACTThe GMAC Validity Study Service was created to enable graduate schools of management to investigate the adequacy of the devices and procedures they use for selection purposes and to facilitate the accumulation and summary of data from many individual graduate management schools. Participating schools are required to provide information on three standard predictors (GMAT‐V, GMAT‐Q, and undergraduate grade point average) and one standard criterion (first year average in management school). In addition, each school may specify optional subgroupings, predictors, and criteria relevant to its admissions process. In the 1979–80 academic year, 38 schools participated in the Validity Study Service. This report provides (a) a descriptive summary of the validity of GMAT scores and undergraduate grade point average as predictors of first year grade point average in management school, (b) a categorization of optional variables designated by individual schools along with the correlational analyses for the most frequently specified variables, and (c) a comparison of the results of the 1979–80 Validity Study Service with the results from previous years.

A SUMMARY OF RESULTS OF THE GRADUATE MANAGEMENT ADMISSION COUNCIL (GMAC) VALIDITY SERVICE FOR 1979

ABSTRACTThe 1978–79 academic year marked the first fully operational year of the Graduate Management Admission Council (GMAC) validity study service. A summary of results is reported for 25 graduate management schools that participated in this first operational series of studies. Results include correlations of standard preadmission measures (GMAT scores and undergraduate grades) with a standard criterion (first‐year average) for all schools. Additional analyses based on school‐designated subgroups and on additional nonstandard preadmission measures and criteria are reported for schools that provided those optional data. The results indicate the potential value of each of various preadmission indices for predicting academic success in graduate management schools.

RELATIONSHIPS OF PREADMISSION MEASURES TO ACADEMIC SUCCESS IN GRADUATE MANAGEMENT EDUCATION

ABSTRACTAs part of the development of a validity study service for the Graduate Management Admission Council, validity studies were conducted for 10 graduate schools of management. Results are summarized for all 10 schools in terms of the effectiveness of standard predictors (GMAT scores and undergraduate grades) for predicting a standard criterion (first‐year average). Analyses based on school‐designated subgroups and on additional nonstandard predictors and criteria are also reported for schools providing those data. The results indicate the value of each of the various preadmission indices for predicting academic success in graduate management schools.

COMPARISON OF A BAYESIAN AND A LEAST SQUARES METHOD OF EDUCATIONAL PREDICTION1

ABSTRACTThe prediction systems under discussion apply where the following conditions obtain: Predictor data are given on the same scale, criterion scores may be given on different scales, and it is necessary to pool data even though criterion scale differences exist. Such a system may be needed for minority group or graduate student prediction where the group sizes are small. Least squares and Bayes methods are used in a cross‐validation study conducted for comparison purposes. Data for the study were taken from the files of the Validity Study Service of the College Entrance Examination Board. A very limited amount of data were supplied by a few American graduate schools. The Bayes method was better, but it was found that both methods yield negative regression weights; when the absolute values of the weights were used, the methods were both improved and yielded results which were very similar in terms of evaluative statistics computed in the cross sample.