THE INTERNAL CONSTRUCT VALIDITY OF THE SAT1

Abstract

ABSTRACTThe primary goal of this research was to obtain a fuller understanding of what is being measured by the SAT. In addition, this project had two secondary goals: (1) the identification of a set of analytic techniques that can satisfactorily assess the dimensionality of the SAT; and (2) the identification of components of a cost‐effective and informative system for monitoring the internal construct validity of the SAT.The principal project findings are: The SAT‐Mathematical test is effectively unidimensional. There is little empirical justification for dividing the total SAT‐Mathematical score into subscores on the basis of item content related to subject area. For purposes of differential item functioning analysis, the total SAT‐Mathematics score should be an adequate matching variable. The SAT‐Verbal test is composed of two distinct albeit highly‐related dimensions, a reading dimension and a vocabulary dimension. Hence, there is an empirical justification for reporting two separate subscores. Currently, antonym and analogy items count toward a vocabulary subscore; sentence completion and reading comprehension items count toward a reading subscore. The empirical data indicate that the sentence completion items appear to belong more with the analogy and antonym items than with the reading comprehension items. The LISREL analyses also provide some evidence for separating the Vocabulary dimension into two dimensions, Antonyms and Analogies. The evidence is inconsistent, however. For the purposes of differential item functioning analysis, it may be necessary to match on the current Reading subscore (reading and sentence completion) for reading comprehension items, an analogy/sentence completion criterion for analogies, an antonym/sentence completion criterion for antonyms and any of these three criteria or SAT‐Verbal for sentence completion items. This research has several findings that have implications for redesigning the SAT‐Verbal test. First, the sentence completion item type is the item type that is most like the total SAT‐Verbal score in what it measures, while the reading passage items are measuring the construct that is least like the total SAT‐Verbal score. Second, speededness effects on SAT‐Verbal may be large enough to produce a factor in item level data that is more salient than a general reading passage factor. Third, the analogy item seems to be the least reliable item type, while the sentence completion item type appears to be the most reliable. This array of results points in the direction of using more reading passage items to achieve a more distinct and reliable reading subscore, using more sentence completion items because they are more reliable and may take less time than other items, and using fewer analogy items because they are less reliable and probably more time consuming than sentence completions or antonyms. The analyses conducted in this study have served to underscore the value of using confirmatory factor analysis of item parcel data to study the dimensional structure of test data. This approach is computationally inexpensive, and appears to provide meaningful and consistent results. The use of parallel parcels makes item data amenable to linear factor analysis. The method seems to circumvent the problems associated with directly factor analyzing item data, namely, the propagation of artifactual “difficulty” factors. The method can also avoid the problem of observing a “speed” factor, as items from later positions in the test can be balanced across parcels. In sum, the parallel parcel approach can be used to dispense with difficulty and speed factors and, hence, obtain a clearer look at the substantive factor structure of the test. Full information factor analysis is not ready for routine use with ATP data. The TESTFACT program is very expensive and sometimes yields “methodological” factors under the full information mode. The TESTFACT program can also produce a least squares factor analysis of a smoothed positive definite matrix of tetrachorics corrected for guessing at a fraction of the cost of the full information solution. Unfortunately, this solution appeared to yield difficulty factors for both SAT‐Verbal and SAT‐Mathematical.