NONRESPONSE BIAS IN THE GRE BACKGROUND QUESTIONNAIRE

Abstract

ABSTRACTResults of some studies of the GRE database suggest that analyses relying upon information from the background questionnaire may lead to misleading or erroneous conclusions because of nonresponse bias. This study examined existing data from all examinees who took the General Examinations for the first time in 1984/85 to estimate the extent of non‐response bias and how seriously it may be affecting statistical summaries and interpretations.Results indicate that nearly all examinees who answer the first item on the questionnaire generally complete the questionnaire or omit just one or two items. However, 16% never answer the first item.Of those who indicate on the first item that they are taking the GRE for the first time, the response rate to all other items is generally better than 97%. The response rate is even higher for U.S. citizens and resident aliens.The first five questionnaire items involve various branching instructions, and examinees make a number of errors in the course of branching. U.S. citizens and resident aliens are instructed to answer the questions on ethnicity and state of residence, while all others are to skip these two questions. Almost 98% of the U.S. citizens answer the ethnicity question, but 17% of the foreign examinees living abroad answer the ethnicity question when they have been instructed to skip it.There is no correlation between GRE scores and the number of questionnaire items left blank. However, the differences in test score averages between examinees who omit a given question and examinees who answer it are often quite large. Furthermore, those who omit a question may have either higher or lower average test scores than those who answer it. Foreign examinees are generally more likely to omit a question than are U.S. citizens. Among U.S. citizens, those who answer a question often have higher GRE quantitative and analytical scores, but lower verbal scores, than those who omit the question.Despite the sometimes large differences in test scores between those who omit an item and those who answer it, the numbers of omitters are small–generally on the order of 1 or 2%. The extent to which their test scores affect statistical summaries of the much larger group averages is negligible.This study concludes that there appears to be some confusion regarding the branching instructions involving the first five items, with the result that some foreign examinees answer the ethnic question when they should not. Because most statistical summaries are broken down by citizenship and ethnicity, these questions are important. A recommendation is made to simplify these items future revisions of the questionnaire by removing branching instructions.