Test Of Standard Written English Research Articles

THE EFFECTS OF SAMPLE RESTRICTION ON GENDER DIFFERENCES*

THE EFFECTS OF SAMPLE RESTRICTION ON GENDER DIFFERENCES*

STUDENT GROUP DIFFERENCES IN PREDICTING COLLEGE GRADES: SEX, LANGUAGE, AND ETHNIC GROUPS

ABSTRACTPart 1 of this study investigated possible causes of the observed decline in correlations between SAT scores and freshman grade‐point average (FGPA). The results were described in Chapter 12, “Implications of Using Freshman GPA as the Criterion for the Predictive Validity of the SAT,” and were the basis for much of Chapters 2 and 3 of the monograph Predicting College Grades: An Analysis of Institutional Trends Over Two Decades (Willingham, Lewis, Morgan, and Ramist 1990). Working with a data base of 38 colleges, the study found that the comparability of course grades received by entering freshmen declined in the 1980s. Three new measures of grade comparability—variety of courses taken, variation in average student aptitude among courses, and appropriateness of average course grade in relation to student aptitude level—proved to be excellent indicators of both the level of and the change in SAT validity for predicting FGPA among the 38 colleges. Using course grade as the criterion instead of FGPA reduced the decline in both SAT and high school GPA (HSGPA) validity for predicting course grades by 40 percent. Contrary to the assumption that high school record (HSR) is a better predictor than the SAT, compared with HSR the SAT had higher or equal average validities for predicting course grade in almost all categories of courses. (Each course was placed into one of 37 categories based on subject, skills required, and level.)Part 2 of this project examines course selection, grading patterns, grade comparability, SAT predictive effectiveness, and average over‐ and underpredictions in each type of course for groups defined by an academic composite index, sex, English as best or not best language, and ethnic group. SAT predictive effectiveness is determined with and without HSR on the basis of correlations that are corrected for restriction of range. Over‐ and underpredictions are determined by residuals from predictions. All results are analyzed by college selectivity level and size.On average, males took more rigorously graded courses and females obtained a higher FGPA: two‐thirds of the .09 difference by sex in FGPA related to course selection. Predictions of course grades based on the SAT were better for females, on average, than for males, and the SAT added more incremental information over HSR for females. Underprediction of FGPA for females, using the SAT and HSR, averaged .06. Underprediction of course grade for females, using the SAT and HSR, averaged .03, but was reduced to .02 using the Test of Standard Written English (TSWE) as an additional predictor, and was eliminated entirely at more selective colleges.Although on average the SAT predicted FGPA and course grades better for students whose best language was English, it added more incremental information over HSR for students whose best language was not English. Asian American students took, on average, very strictly graded courses, but obtained a high average FGPA. The SAT predicted FGPA and course grades better for them than for any other ethnic group. On average, the SAT added more incremental information over HSR in predicting FGPA and course grades for black students than for any other ethnic group. Course grades were the least comparable for Hispanic and black students. They were so lacking in comparability for Hispanic students that, on average, there was better prediction of one course grade, as long as the course was identified, than of FGPA, even though the latter was typically based on eight or nine courses (for black students, course grade and FGPA predictions were equally good).On average, substantial improvement in the prediction of FGPA could be obtained by using, as an additional predictor, the average grading difficulty of courses selected by students. The improvement was greatest at less selective colleges and for students of lower academic levels.The highest average correlations in predicting FGPA were obtained by predicting each of a student's course grades separately and averaging the predictions to obtain a predicted FGPA. After correcting for predictor restriction of range and criterion unreliability, the average correlations were .64 for the SAT, .67 for HSR, and .75 for the multiple correlation of the SAT and HSR. These correlations may be the best estimates ever made of the effectiveness of the SAT and HSR for predicting FGPA, because they were based on a large cross section of colleges and on comparable grades for all courses taken by a student, and because the effects of both predictor restriction of range and criterion unreliability were removed.

CHARACTERISTICS ASSOCIATED WITH DIFFERENTIAL ITEM FUNCTIONING ON THE SCHOLASTIC APTITUDE TEST: GENDER AND MAJORITY/MINORITY GROUP COMPARISONS

ABSTRACTThe purpose of the study was to investigate whether selected test and item characteristics in the SAT are associated with unexpected differential item functioning (DIF) for males and females and for majority and minority group members (i.e., White performance compared with Black, Asian‐American, Hispanic, and American Indian performance). Six forms of the SAT that were administered relatively recently were used. Each of the six forms consists of Verbal sections (containing Analogies, Antonyms, Sentence Completions, and Reading Comprehension), Mathematics sections (containing Regular Mathematics problem‐solving and Quantitative Comparisons), and the Test of Standard Written English (TSWE) (containing Usage and Sentence Correction). Findings from previous studies, test specifications, and suggestions offered by test development experts led to the identification of more than one hundred a priori item coding categories, and each SAT item was coded accordingly. Items were coded with regard to type, content, and format.The Mantel‐Haenszel procedure was used to provide an index of differential item functioning (DIF) for each reference/focal group comparison. Females, Asian‐Americans, Blacks, Hispanics, and American Indians were the focal groups; males and Whites were the reference groups. With the exception of the American Indians, each focal group was represented in sufficient numbers on each test form to lead to meaningful interpretation of data. For each item category, one‐way analyses of variance were computed using as the dependent variable the Mantel‐Haenszel DIF values. Analyses were run separately for each reference/focal group investigated.The study reports on patterns of differential performance across SAT sections as well as on section‐specific differences. Patterns across several ethnic groups and between two gender groups, as well as results specific to each of the groups, are identified. The report addresses the following questions:• Are there unexpected group differences associated with the points tested (e.g., percentages, verb forms)?• What aspects of item content (e.g., subject matter, gender and ethnic references, level of language) are associated with unexpected group differences?• Are there elements of test or item format (e.g., length of stem, formatting or location of the stem or options) that are associated with unexpected group differences?

A STUDY OF GENDER AND PERFORMANCE ON ADVANCED PLACEMENT HISTORY EXAMINATIONS

ABSTRACTSeveral studies have shown that, on average, women perform slightly better than men on free‐response tests, while men perform slightly better on multiple‐choice tests, Studies of the College Board Advanced Placement (AP) Examinations have revealed a similar phenomenon. For almost all AP Examinations, men average better on both parts of the tests, but sex differences on the free‐response parts are almost always less, and for some tests they are nonsignificant.Two AP Examinations, U.S. History and European History, were selected for study because sex differences on the free‐response parts were nonsignificant while sex differences on the multiple‐choice parts were large. Random samples of free‐response booklets were drawn from the 1986 administrations of both examinations. A number of ratings and analyses were made of the responses: English composition quality, historical content, responsiveness, factual errors, handwriting quality, neatness, and number of words written. All variables were then used to predict the free‐response scores. Several significant predictors were observed: the AP multiple‐choice score, historical content, English composition quality, and the number of words written.Predictor variables were classified with respect to their degree of relevance to the measurement construct. The historical content ratings and multiple‐choice history scores were considered to be directly relevant, English composition quality and words written to be indirectly relevant, and neatness and handwriting to be irrelevant. For the two free‐response scores combined, the indirectly relevant variables, in combination, predicted about as well as the directly relevant variables. For the Part A free response, the indirectly relevant variables predicted best, while for the Part B free response, the directly relevant variables predicted best. Handwriting and neatness did not predict free‐response scores well at all.The regression equations developed were also used to estimate over‐ or underprediction by sex of free‐response scores from the AP multiple‐choice score alone and from multiple predictors. As expected, the AP multiple‐choice score underpredicted female performance on the free‐response part of the test and overpredicted male performance. For the AP U.S. History Examination, the over/underprediction effect size was reduced to a nonsignificant level when the English composition quality of the free responses was added to the predictor set. For European History, the over/underprediction effect size was also reduced to a nonsignificant level by the addition of English composition quality, although the reduction of the effect size was not as dramatic as that for U.S. History. No other predictor variables contributed importantly to the reduction of effect sizes.To develop further insight into this test format phenomenon, AP files for the sampled cases were matched with College Board Admissions Testing Program (ATP) data to obtain Scholastic Aptitude Test (SAT) scores and English Composition Test scores. In U.S. History, the SAT‐verbal, Test of Standard Written English (TSWE), and English Composition Test (ECT) scores made significant independent contributions to the prediction of AP free‐response scores over and above that possible using the AP objective (multiple‐choice) score, especially for Part A. In European History, SAT‐verbal scores made significant independent contributions, although TSWE and ECT scores did not.This study suggests that format effects are real and cannot be attributed to bias in scoring or to totally irrelevant variables. When scoring was conducted analytically with a focus on historical content, no sex differences were observed in the free‐response portions. This is the same result observed for the regular administration readings, which are graded holistically and by readers different from those used for this study. The sexes appear to differ in how they respond to two legitimate but different ways of assessing history skills. The study raises the question of how much influence basic skills and aptitude should have on the outcomes of assessments in specific areas of achievement.

COHORT DIFFERENCES ASSOCIATED WITH TRENDS IN SAT® SCORE AVERAGES

ABSTRACTThe potential impact of cohort changes with regard to ethnic group, gender, class rank, and first language learned on yearly average Scholastic Aptitude Test (SAT) and Test of Standard Written English (TSWE) scores was studied. Regression and cross‐classification analyses were conducted on data from 1985, and 1987 to 1990. Both analyses adjusted test score means based on common cohorts of students. The analyses indicated that approximately half of the decline in SAT verbal (SAT‐V) scores was associated with cohort change. Average SAT mathematics (SAT‐M) scores could possibly have risen by three points, rather than remaining constant, if the 1987 and 1990 cohorts were the same.

A Closer Look at the Advanced-Standing Program in Social Work Education

The abbrevtated advanced-standing program in social work education is evaluated in light of evidence that Bachelor of Social Work as compared to non-Bachelor of Social Work degree applicants are substantially less prepared to pursue graduate studies. Except for a few applicants from prestige colleges, discrepancies between the undergraduate grade potnt average and scores on the Graduate Record Examination and Test of Standard Written English reveal the existence of undergraduate grade inflation in most public and private institutions of higher education. A method for discounting the undergraduate grade point average is presented. Adopting an admissions policy that sets a higher graduate record examination or discounted undergraduate grade point average standard for acceptance into the advanced-standing program is recom mended by virtue of the unavoidable tradeoff between quantity and quality.

SAT TESTING TIME FOR STUDENTS WITH DISABILITIES

ABSTRACTThis study examined the amount of time used by disabled students to take special administrations of the SAT offered by the College Board's ATP Services for Handicapped Students. It also investigated the relationship between length of time taken and scores achieved.Timing data on groups of students were examined, both by disability and by the version of the test taken (braille, cassette, large type or regular type). Results showed that disabled students who were tested over the four years from 1979 to 1983 took considerably more time to complete the SAT than non‐disabled students. Hearing impaired students taking a regular version of the test took the smallest amount of time, averaging 3.7 hours, while visually impaired students taking a cassette version averaged 5.4 hours and those taking a braille version averaged 6.0 hours. Physically disabled and learning disabled students averaged between 4.3 and 4.7 hours.Time spent on each section of the SAT (two verbal sections, two math sections and the Test of Standard Written English) was also investigated, although those numbers did not differ substantially from total timing data.The relationship between SAT test scores and the amount of time spent on the verbal and math section of the test was investigated as well. Results showed that physically disabled and visually impaired students who spent more time on the verbal sections of the SAT achieved lower verbal scores than those who spent less time, while the opposite was true for the verbal scores of hearing‐impaired and learning‐disabled students. However, for the math sections of the SAT, students in all disabled groups who spent more time tended to achieve higher math scores.

MEASURING THE ORGANIZATIONAL ASPECTS OF WRITING ABILITY

The present study assessed the relationship among holistic writing ability, the Test of Standard Written English (TSWE), and the following tests of organizational ability: anagram solving, word reordering, sentence reordering, and paragraph assembly. Based upon a sample of 105 undergraduate students, the main findings were that writing ability, as measured by the holistic method of scoring, was significantly correlated with performance on the TSWE and the four tests of organizational ability. A composite score on all four organizational tests was found to have the highest zero‐order correlation with the measure of writing ability. A stepwise regression analysis, with the measure of writing ability as the criterion, also indicated that the composite score explained a significant proportion of the variance beyond that explained by the TSWE. The results are discussed in terms of the Kintsch and van Dijk model of strategic discourse processing, which suggests that different organizational strategies operate at the levels of words, sentences, and paragraphs. It is concluded that tests assessing organizational strategies ought to be included in assessments of writing ability.

The Effects of Instructors and Student Myers-Briggs Type Indicator Characteristics on the Accuracy of Grades Predicted for an Introductory English Composition Course

The effects of different instructors and student Myers-Briggs Type Inventory personality characteristics on the accuracy of predicting 1041 introductory English composition grades from a single regression equation were investigated. The instructor variable was defined by three groups of classes: high grade means, average grade means, and low grade means. The Scholastic Aptitude Test Verbal, Test of Standard Written English, and high school class percentile rank were used as the predictors. Analyses compared the regression equations of the class and student personality subgroups. The results suggested that grades were inflated in classes with high means and deflated in classes with low means. For these classes, there was a difference between the regression equations and there was an interaction of the equations with student personality. In the average class, there was not a difference among the student personality regression equations, although grades were more predictable for certain personality types than others. The findings indicate that the accuracy of predicted grades can be substantially influenced by differences among instructors and student personality types.

Intercorrelations of Measures of Reading and Writing: Implications for College Research on Reading and Writing

64 college students participated in a study to examine the intercorrelationships of reading and writing measures which included The Nelson-Denny Reading Test, Form E, a randomly deleted cloze test, and the Test of Standard Written English. The latter correlated highly with the cloze and The Nelson-Denny tests, each contributing about an equal amount of common variance. The common variance between The Nelson-Denny and cloze was substantially less, so it was concluded that research at the college level should utilize both cloze and standardized reading tests to investigate the effect of sentence-combining on reading comprehension.

Subject Matter Experts' Assessment of Item Statistics

The study was conducted to determine the degree to which subject matter experts could predict the difficulty and discrimination of items from the Test of Standard Written English. Two studies were conducted. The first study was meant to serve as a training exercise while the second study was meant to simulate the conditions under which ratings would be obtained in practice. It was concluded that despite the extended training period the raters did not approach a high level of accuracy, nor were they able to pinpoint the factors that contribute to difficulty and discrimination. Further research should attempt to uncover those factors by examining the items from a linguistic and psycholinguistic perspective. It was argued that by coupling linguistic features of the items with subject matter ratings it may be possible to attain more accurate predictions of difficulty and discrimination.

A Study of Pre-Equating Based on Item Response Theory

The study reports a feasibility study using item response theory (IRT) as a means of equating the Test of Standard Written English (TSWE). The study focused on the possibility of pre-equating, that is, deriving the equating transformation prior to the final administration of the test. The three-parameter logistic model was postulated as the response model and its fit was assessed at the item, subscore, and total score level. Minor problems were found at each of these levels; but, on the whole, the three-parameter model was found to portray the data well. The adequacy of the equating provided by IRT procedures was investigated in two TSWE forms. It was concluded that pre-equating does not appear to present problems beyond those inherent to IRT-equating.

ESTIMATING THE EFFECTS OF VARIOUS METHODS OF PREPARING FOR THE SAT

ABSTRACTA national sample of Scholastic Aptitude Test (SAT) candidates was surveyed to determine the ways in which they had prepared for the examination. Candidates' SAT scores were retrieved from test files, along with selected background information that included rank in high school class and scores on the Preliminary Scholastic Aptitude Test (PSAT), which most students had taken about eight months before they took the SAT.Multiple regression analyses revealed effects of less than ten points on the 200‐800 SAT score scale and less than two points on the 20–60 score scale for the Test of Standard Written English (TSWE) for each of the methods of test preparation. A significant number of negative effect estimates, coupled with significant differences between the background characteristics of preparing and nonpreparing groups, strongly suggested the influence of selection bias in the estimates. Exploratory analyses of the effects of combinations of methods of preparation showed that various methods may interact in complex ways with other methods and with some student traits. Furthermore, self‐selection may not be uniform across levels of ability and previous achievement, and may also depend on the candidates' selection of other methods of preparation. The implications of the results for the design of further research on the effects of special preparation for tests are discussed.

A Re-Examination of the Use of the Test of Standard Written English and Resulting Placement for Older Upper-Division and Master's Level Students

The Test of Standard Written English (TSWE) was developed to assess the ability of college students to use English. Most of the research has validated the use of the TSWE with freshman level students. The one study with older junior and master's degree level students was replicated with the one-year cumulative grade point average serving as the criterion variable. Confirmed was the finding that TSWE scores, resulting placement, and subsequent success in an English course (if required) was predictive of the academic success of older upper-division and master's level students.

A COMPARISON OF PROCEDURES TO ASSESS WRITTEN LANGUAGE SKILLS AT GRADES 4, 7, AND 10

Language Arts is a major area of instruction in the nation's schools. This area focuses on the structure and use of the English language, and serves as a foundation for writing skills. Whether or not schools are succeeding in training students to write adequately is currently a matter of great controversy and concern. The assessment of written language skills at all levels of educational achievement, which is related to this concern, has recently received much attention. Procedures to assess written language skills are typically divided into two dichotomous categories: direct procedures and indirect procedures. Direct assessments require a sample of writing from an examinee and a response from a human evaluator; they are generally thought of as Indirect assessments typically involve a series of multiple-choice, machine-scorable items, and are generally called objective tests. The standard line of inquiry into writing assessment procedures compares the performance of student writers on essay tests and objective tests. Moderate correlations between these two assessment procedures have been repeatedly found with college-level or near-college-level students. For example, Breland, Conlon, and Rogosa (1976) found a correlation of .42 for a group of 96 college freshmen between scores on the College Board Test of Standard Written English (TSWE), a 50-item multiple-choice test, and a 20-minute essay scored on a four-point scale with two independent ratings. A second study by Breland and Gaynor (1979) reported correlations for college freshmen of .63 (n = 819), .63 (n = 895) and .58 (n = 517) between TSWE scores and essays scored on a six-point scale by two raters. The correlation involving the sum of the three essay scores with the sum of the three TSWE scores was .76. A similar study conducted by Huntley, Schmeiser, and Stiggins (1979) compared scores obtained from 3 samples of approximately 50 college students on the American College Testing Program's English Usage Test (a 75-item, multiple-choice test), and the sum of three essays, each of which was scored by two raters. The correlations obtained were .43, .50, and .67. In a now classic study of the performance of 646 junior and senior high school students, Godshalk, Swineford, and Coffman (1966) reported correlations of from .46 to .71 between the sum of 5 essay scores, each score produced by 5 independent ratings and various subtests of the College Board English Composition Test (ECT). They also found correlations ranging from .72 to .75 between the composite essay score and a score produced by various groupings of three ECT subtests. A recent study by Hogan and Mishler (1980) extended this inquiry to the elementary school level. Hogan and Mishler gave the Metropolitan Achievement Test-Language Instructional Tests (MAT-LIT) and 20-minute essay tests to approximately 140 children in Grade 3 and 160 children in Grade 8. Approximately 60 students at each grade level

SUBJECT MATTER EXPERTS' ASSESSMENT OF ITEM STATISTICS

ABSTRACTThe study was conducted to determine the degree to which subject matter experts could predict the difficulty and discrimination of items from the Test of Standard Written English. Two studies were conducted. The first study was meant to serve as a training exercise while the second study was meant to simulate the conditions under which ratings would be obtained in practice. It was concluded that despite the extended training period the raters did not approach a high level of accuracy, nor were they able to pinpoint the factors that contribute to difficulty and discrimination. Further research should attempt to uncover those factors by examining the items from a linguistic and psycholinguistic perspective. It was argued that by coupling linguistic features of the items with subject matter ratings it may be possible to attain more accurate predictions of difficulty and discrimination.

The Test of Standard Written English and Resulting Placement Patterns: a Follow-Up of Performance of Older Upper-Division and Master Level Students

The Test of Standard Written English (TSWE) was designed for assessing the ability of students to use English. Published research has demonstrated that the TSWE has predictive validity for grades earned in freshman English and that the TSWE can be used for appropriate placement of students in the freshman English curriculum. Results of this validity study with the TSWE suggest that the use of the instrument can be generalized to older students entering at the juniors and master's degree level. The TSWE scores, resulting placement, and subsequent success in an English course, if required, were found to be predictive of academic performance for older, upper-division and masters level students in an environment with a restricted grading system.

EFFECTS OF TAKING THE SAT ON RATE OF WORK AND GUESSING

ABSTRACTThe College Board recently published Taking the SAT, a booklet designed to Improve candidates' familiarity with the Scholastic Aptitude Test (SAT) and the Test of Standard Written English (TSWE). This note examines the extent to which receiving the new booklet altered (a) candidates' strategies with regard to responding in the face of uncertainty and (b) the rate at which examinees proceeded through the test. It Is suggested that specific instructions found in the booklet are related to changes in examinees' test‐taking behaviors in both of these areas.

RELATIONSHIPS BETWEEN ESSAY TESTS AND OBJECTIVE TESTS OF LANGUAGE SKILLS FOR ELEMENTARY SCHOOL STUDENTS

Interest in the relationship between essay tests and objective tests has been a persistent one in educational measurement. Research on this relationship may be divided into two broad categories. First, there are those studies which compare the relative effectiveness of the essay and the objective tests as measures of knowledge of content, such as science, history, etc. Coffman's (1971) excellent review of the literature concentrates primarily, although not exclusively, on this area. Second, there are those studies which compare essays and objective tests as measures of writing skill. While a substantial amount of research has been devoted to this latter issue, significant questions have yet to be answered-or even raised-and the questions are becoming more critical in light of demands for competency testing and criticisms of all objective tests used to assess writing skill (see, for example, National Council of Teachers of English, 1975). Research has been particularly scarce below the college entrance level. Traditionally, most research regarding essays versus objective tests as measures of writing skill has been related to college selection and/or placement and, hence, has dealt with the higher developmental levels of writing skill. For a large sample of college freshmen, Breland and Gaynor (1979) report correlations between single essays and single scores on the objective (multiplechoice) Test of Standard Written English of .56-.63; see also Breland (1977). The widely cited study by Godshalk, Swineford, and Coffman (1966), using a largely college-bound group of high school juniors and seniors reported correlations generally in the .30's between several objective measures and single essays rated by two or three readers. But correlations of .57 to .71 were obtained between objective measures and an elaborately constructed essay score (four writing samples, each scored by five readers). Major attempts to measure writing skill at younger ages, specifically at ages 9 and 13, have been made by National Assessment of Educational Progress (1972, 1975). However, because of the special nature of the NAEP effort, no correlations between essay-type and objective-type measures of writing skill have been reported from this

Using Longitudinal Data to Estimate Reliability in the Presence of Correlated Measurement Errors

Test-retest correlations can lead to biased reliability estimates when there is instability of the true scores in the interval between tests and/or when the measurement errors are correlated. Using three occasion data on the Test of Standard Written English and essay ratings, an analysis is demonstrated which separates true score instability and correlated errors.