Trial State Assessment Research Articles

Synthesizing Results from the Trial State Assessment

Using data collected under the Trial State Assessment (TSA) of the National Assessment of Educational Progress (NAEP), this article describes and illustrates a two-stage statistical model for investigating state-to-state variation in mathematics achievement. At the first stage, within each state, a two-level hierarchical linear model is estimated via maximum likelihood. At the second stage, results are combined across states using Bayesian estimation implemented via the Gibbs sampler. The results reveal considerable state-to-state heterogeneity in mathematics proficiency, but most heterogeneity is explainable on the basis of covariates defined on students, teachers, and schools. The findings suggest that interest in state comparisons might productively focus on state differences in policy-relevant correlates of proficiency rather than on state differences in mean proficiency. The analytical approach can be applied in other cases where data are dense at the lower level of a hierarchy but thin at the higher level.

Applications and Extensions of MCMC in IRT: Multiple Item Types, Missing Data, and Rated Responses

Patz and Junker (1999) describe a general Markov chain Monte Carlo (MCMC) strategy, based on Metropolis-Hastings sampling, for Bayesian inference in complex item response theory (IRT) settings. They demonstrate the basic methodology using the two-parameter logistic (2PL) model. In this paper we extend their basic MCMC methodology to address issues such as non-response, designed missingness, multiple raters, guessing behavior and partial credit (polytomous) test items. We apply the basic MCMC methodology to two examples from the National Assessment of Educational Progress 1992 Trial State Assessment in Reading: (a) a multiple item format (2PL, 3PL, and generalized partial credit) subtest with missing response data; and (b) a sequence of rated, dichotomous short-response items, using a new IRT model called the generalized linear logistic test model (GLLTM).

Synthesizing Results from the Trial State Assessment

Synthesizing Results from the Trial State Assessment

Projecting to the NAEP Scale: Results from the North Carolina End‐of‐Grade Testing Program

Data from the North Carolina End‐of‐Grade test of eighth‐grade mathematics are used to estimate the achievement results on the scale of the National Assessment of Educational Progress (NAEP) Trial State Assessment. Linear regression models are used to develop projection equations to predict state NAEP results in the future, and the results of such predictions are compared with those obtained in the 1996 administration of NAEP Standard errors of the parameter estimates are obtained using a bootstrap resampling technique.

Inequality of Access to Educational Resources: A National Report Card for Eighth-Grade Math

This article considers social and ethnic inequality in access to resources for mathematics learning in eighth grade: favorable school disciplinary climate, advanced course offerings, teacher subject-matter preparation, and emphasis on reasoning during classroom discourse. Data are from 41 states and territories 1 participating in the 1992 Trial State Assessment (TSA) of the National Assessment of Educational Progress (NAEP). Socially advantaged students typically had greater access to these resources than did socially disadvantaged students. Access also depended on student ethnicity. However, the degree of social and ethnic inequality in access varied significantly across states. New methods for assessing and displaying state-to-state variation in social and ethnic inequality are illustrated. We argue that “report cards” displaying state differences in student proficiency are, by themselves, misleading; state differences in access to key educational resources provide an important supplement.

IMPLEMENTING SHAFFER'S MULTIPLE COMPARISON PROCEDURE FOR LARGE NUMBERS OF GROUPS

ABSTRACTShaffer (1986) presented a multiple comparison procedure for paired comparisons that had superior power to many available alternatives. Unfortunately, use of the procedure has been severely limited by the complexity of its implementation. This paper presents a method of allowing Shaffer's procedure to be used in a much larger class of problems.Theoretical results concerning an aspect of Shaffer's procedure are derived. The use of these results in implementing the procedure is the described. Next, two heuristics are described which greatly enhance the efficiency of the method. Finally, an efficient algorithmic method of implementing Shaffer's procedure is outlined.The present work allows Shaffer's procedure for multiple comparisons to be applied to large numbers of groups. Unlike earlier work, no assumptions about the testing situation need be made. These methods are illustrated by application to two data sets; comparisons of the order of 11 clustering methods, and comparison of the means of 44 states from the 1994 National Assessment of Educational Progress Trial State Assessment of Reading at Grade 4.

Notes: A Study of the Utility of Results From the 1992 Trial State Assessment (TSA) in Reading for State-Level Administrators of Assessment

This study investigated the usefulness of the 1992 TSA in reading for state-level administrators of assessment. Data regarding their perceptions about the credibility and orientation of various components of the TSA were collected (using phone interviews) from 26 state directors of assessment, or their designees, and analyzed. Additionally, data about the utility of the manner of reporting TSA results, how results are currently used and could be improved, were collected and analyzed. The following four conclusions were drawn from the study: involve the International Reading Association and more teachers, continue but modify descriptors, disseminate results quicker, and make results more useful.

A Study of the Utility of Results from the 1992 Trial State Assessment (TSA) in Reading for State-Level Administrators of Assessment

A Study of the Utility of Results from the 1992 Trial State Assessment (TSA) in Reading for State-Level Administrators of Assessment

Inequality of Access to Educational Resources: A National Report Card for Eighth-Grade Math

This article considers social and ethnic inequality in access to resources for mathematics learning in eighth grade: favorable school disciplinary climate, advanced course offerings, teacher subject-matter preparation, and emphasis on reasoning during classroom discourse. Data are from 41 states and territories1 participating in the 1992 Trial State Assessment (TSA) of the National Assessment of Educational Progress (NAEP). Socially advantaged students typically had greater access to these resources than did socially disadvantaged students. Access also depended on student ethnicity. However, the degree of social and ethnic inequality in access varied significantly across states. New methods for assessing and displaying state-to-state variation in social and ethnic inequality are illustrated. We argue that “report cards” displaying state differences in student proficiency are, by themselves, misleading; state differences in access to key educational resources provide an important supplement.

The Use of a Person-Fit Statistic with One High-Quality Achievement Test

After analyzing data from the 1990 National Assessment of Educational Progress Trial State Assessment, we question whether person-fit statistics are useful in the analysis and reporting of results from psychometrically strong achievement tests. Using a weighted mean-square person-fit statistic, we examined the distribution of fit across individuals, looked for group and item-type differences, and investigated practical significance. In each analysis, we found that this person-fit statistic did not provide any additional information.

Linking statewide Tests to the National Assessment of Educational Progress: Stability of Results

The adequacy of linking statewide standardized test results to the National Assessment of Educational Progress (NAEP) by using equipercentile equating procedures was investigated. Statewide mathematics test data for eighth-grade students in 1990 and 1992 were obtained from four states. NAEP data for samples from these four states were obtained from the results of the Trial State Assessment administrations in the same years. Equating functions for male and female students in two states providing gender identification were similar at the low end of the scale but diverged at the high end of the scale. Applications of the equating functions obtained for 1990 data to the statewide test results obtained in 1992 provided estimates that were generally similar to actual NAEP results near the median, but not in the tails of the distribution. These results suggest that such linking, although reasonable for estimating average performance for the state, are not sufficiently trustworthy to use for making comparisons ba...

Assessing the Content of the Trial State Assessment of the National Assessment of Educational Progress in Reading

Assessing the Content of the Trial State Assessment of the National Assessment of Educational Progress in Reading

The Design of the National Assessment of Educational Progress

The key features of the design of the National Assessment of Educational Progress (NAEP) are discussed with particular emphasis on the design to be used for the 1992 assessment. An overview of the design and its philosophy are given with a description of the multicomponent solution to the twin requirements of reliably measuring trends in achievement while responding to changing educational priorities and advances in measurement technology. The student sample designs for the National Assessment and the Trial State Assessment are described. The focused‐balanced incomplete block (focused‐BIB) spiraling method of item sampling is discussed and compared with simpler matrix sampling designs. The impact of the NAEP design on the analysis of assessment data is discussed.

Chapter 8: Applications and Extensions of NAEP Concepts and Technology

The National Assessment of Educational Progress (NAEP) has consistently pioneered new assessment methods in conjunction with developing the psychometric methodologies underlying them. Several NAEP developments—such as complex matrix item sampling designs, the introduction of performance-based items in large-scale assessments, vertical scaling, and an intelligent computer system that produces unique assessment reports for participating jurisdictions in the NAEP Trial State Assessment program—are presented in this chapter along with a discussion of their extensions and applications to other current and future assessment projects.

Sampling and Weighting in the National Assessment

This chapter describes procedures for obtaining the National Assessment of Educational Progress (NAEP) student samples used in the national and state assessments and for deriving survey weights for use in the analysis of the survey data. Following the description of general procedures, more detailed discussion is included about several issues that relate to the procedures used. In some cases, these involve procedures that NAEP is actively reviewing and investigating, with a view toward implementing improvements in the future. In other cases, the procedures, although well established in NAEP, involve technical aspects with interesting features not fully described in the available technical reports. Probability sampling techniques are used to select the student samples for the national assessment and for the Trial State Assessment program. The sample designs for these two types of samples depend on each other only in the implementation of procedures to minimize the overlap of the samples of participating schools. For both the national assessment and the Trial State Assessment, the goal of the sample design is to obtain samples of students from which estimates of subpopulation characteristics can be obtained with reasonably high precision while satisfying economic and operational constraints. We will describe the procedures used to obtain the student samples used in the national and state assessments and to derive survey weights for use in the analysis of the survey data. Additionally, we will discuss several