Test Scores in the Long Run

Abstract

IN A NUMBER of states, students are promoted or allowed to graduate based on their scores on tests. In some states, teachers and administrators are punished if the scores of their students are too low and rewarded if the scores are high. Against this backdrop stands a body of literature that strongly implies that, in the long run, test scores don't predict much of anything. Even the SAT, created by the College Entrance Examination Board to predict who will succeed in college, doesn't make very accurate predictions. The typical correlation between SAT scores and freshman grades is about .45. (The correlation is about the same for high school grades, except in the case of highly selective colleges, where grades have less predictive power because nearly all students arrive with very good high school grades.) Now, the amount of variance in one variable that is predicted by another is given by the square of the correlation. If we square .45, we get .2025. Rendering this proportion as a percentage, we find that the SAT accounts for only about 20% of the variance in freshman grade-point averages. Thus some 80% of what distinguishes who makes the dean's list from who goes on academic probation comes from sources not measured by the SAT. The widespread belief that the SAT is the single determinant of who gets into college - apparently held even by Nicholas Lemann, author of The Big Test: The Secret History of the American Meritocracy - is a triumph of Educational Testing Service marketing over reality. Lemann ought to know better, and college admissions officers do. While researching the matter for a Washington Post review of Lemann's book, I checked the admissions at Brown University. I found that the university could have filled two freshman classes by admitting only applicants who scored between 750 and 800 on the SAT verbal. In fact, the school admitted only one-third of the applicants with these magnificent scores. And it admitted some students with SAT verbal scores as low as 400. College admissions officers know that the SAT is a highly fallible instrument. When Robert Schaeffer of FairTest talks with groups of admission officers, he always asks for a show of hands in response to the question Who would continue to use the SAT if the university had to pay for it? He says that he has yet to see a single arm go up. Unfortunately, the knowledge of these deans of admissions is tempered by the fact that they need to keep SAT averages high in order to show potential alumni donors that Old Ivy U is still doing a great job. When we turn to the workplace, the relationship between test scores and earnings all but disappears. Much of the literature investigating how much test scores contribute to productivity is reviewed by Henry Levin of Columbia University in a paper titled High Stakes Testing and Economic Productivity. The paper can be accessed on the Web at www.law.harvard.edu/groups/civilrights/conferences/testing98/drafts/lev in.html. In studies over the years, an increase in math test scores of one standard deviation has been associated with a 3% to 4% difference in wages. The correlation for reading scores is essentially zero. In the 1980s, the correlation for math scores increased to 7% for males and 14% for females, but no correlation was evident for reading scores. It is true that the National Adult Literacy Survey found that an increase of one standard deviation in reading scores resulted in an 18% difference in income, but this result is suspect and still seems rather small. The result is questionable because literacy and income were assessed simultaneously. It would have been better to assess literacy at one point in time and use those scores to predict income at a second point. When the two are assessed at the same time, reading scores might well be as much a consequence of a job with high literacy demands as a cause of the high wages. Another study found that people who scored high on reading tests also had many more workplace opportunities to practice and hone their literacy skills. …