National Assessment Of Educational Progress Assessment Research Articles

The 2013-15 Decline in NAEP Mathematics: What it Teaches Us about NAEP and the Common Core

ABSTRACT After 25 years with small to moderate gains in performance in mathematics, scores on the National Assessment of Educational Progress (NAEP) main assessment declined between 2013 and 2015 in Grades 4 and 8. Previous research has suggested the decline may be linked to the implementation of the Common Core state standards. In this article, the decline in the NAEP composite score is shown to be driven primarily by losses in the content strands of Geometry and of Data Analysis, Statistics, and Probability. A gain in fractions achievement is also evident in an item-level examination of the NAEP results, but not in reported NAEP scores. These effects are discussed with respect to the CCSS, the rationale for evaluating national progress, and a potential redesign of the NAEP assessment.

Bayesian probabilistic forecasting with large-scale educational trend data: a case study using NAEP

Of critical importance to education policy is monitoring trends in education outcomes over time. In the United States, the National Assessment of Educational Progress (NAEP) has provided long-term trend data since 1970; at the state/jurisdiction level, NAEP has provided long-term trend data since 1996. In addition to the national NAEP, all 50 states and United States jurisdictions participate in the state NAEP assessment. Thus, NAEP provides important monitoring and forecasting information regarding population-level academic performance of relevance to national and international goals. However, an inspection of NAEP trend reports shows that relatively simple trend plots are provided; and although these plots are important for communicating general trend information, we argue that much more useful information can be obtained by adopting a Bayesian probabilistic forecasting point of view. The purpose of this paper is to provide a Bayesian probabilistic forecasting workflow that can be used with large-scale assessment trend data generally, and to demonstrate that workflow with an application to the state NAEP assessments.

Please Don't Use NAEP Scores to Rank Order the 50 States.

Results from the National Assessment of Educational Progress (NAEP) and other large scale tests are often reported as rank ordered lists showing mean values for each of the 50 states. Using data from the 2003 State NAEP Assessment, this paper examines the standard errors associated with State NAEP scores and explains why the use of rank order statistics is inappropriate. An alternate approach anchored to a given state is offered.

Keypresses and Mouse Clicks: Analysis of the First National Computer-Based Writing Assessment

The quality of students’ writing skills continues to concern educators. Because writing is essential to success in both college and career, poor writing can have lifelong consequences. Writing is now primarily done digitally, but students receive limited explicit instruction in digital writing. This lack of instruction means that students fail to take advantage of the affordances of digital tools. The writing process is shaped by the tools used, which makes digital writing, to an extent, different from writing with pen and paper. To better understand students’ digital writing skills, we take advantage of the information provided by computer-based assessments—keyboard and mouse activity data. We examine the relationship between students’ use of the keyboard and mouse during the assessment and students’ writing achievement. Our data comes from the first national computer-based writing assessment in the United States, the 2011 National Assessment of Educational Progress (NAEP) assessment. Using data from over 24,100 eighth-grade students, we found that the number of keypresses had a distinct and direct effect on writing achievement scores, controlling for word count. We also identified several different patterns of keyboard and mouse activity on the computer-based NAEP assessment.

Assessment of fit of item response theory models used in large-scale educational survey assessments

Latent regression models are used for score-reporting purposes in large-scale educational survey assessments such as the National Assessment of Educational Progress (NAEP) and Trends in International Mathematics and Science Study (TIMSS). One component of these models is based on item response theory. While there exists some research on assessment of fit of item response theory models in the context of large-scale assessments, there is a scope of further research on the topic. We suggest two types of residuals to assess the fit of item response theory models in the context of large-scale assessments. The Type I error rates and power of the residuals are computed from simulated data. The residuals are computed using data from four NAEP assessments. Misfit was found for all data sets for both types of residuals, but the practical significance of the misfit was minimal.

The effects of prior computer use on computer-based writing: The 2011 NAEP writing assessment

Writing achievement levels are chronically low for K-12 students. As assessments follow the transition to computer-based writing, differences in technology access may exacerbate students’ difficulties. Indeed, the writing process is shaped by the tools we use and computer-based writing is different from writing with pen and paper. We examine the relationship between reported prior use of computers and students’ achievement on the first national computer-based writing assessment in the United States, the 2011 National Assessment of Educational Progress (NAEP) assessment. Using data from over 24,100 eighth grade students, we found that prior use of computers for school-related writing had a direct effect on writing achievement scores on the computer-based NAEP assessment. One standard deviation increase in prior use led to a 0.14 and 0.16 standard deviation increase in mean and scaled writing achievement scores respectively, with demographic controls and jackknife weighting in our SEM analysis. We also looked at earlier NAEP assessments and found that prior computer use did not positively affect the earlier pen and paper-based writing assessments.

Using response time to investigate students' test-taking behaviors in a NAEP computer-based study

Large-scale survey assessments have been used for decades to monitor what students know and can do. Such assessments aim at providing group-level scores for various populations, with little or no consequence to individual students for their test performance. Students' test-taking behaviors in survey assessments, particularly the level of test-taking effort, and their effects on performance have been a long-standing question. This paper presents a procedure to examine test-taking behaviors using response time collected from a National Assessment of Educational Progress (NAEP) computer-based study, referred to as MCBS. A five-step procedure was proposed to identify rapid-guessing behavior in a more systematic manner. It involves a non-model-based approach that classifies student-item pairs as reflecting either solution behavior or rapid-guessing behavior. Three validity checks were incorporated in the validation step to ensure reasonableness of the time boundaries before further investigation. Results of behavior classification were summarized by three measures to investigate whether and how students' test-taking behaviors related to student characteristics, item characteristics, or both. In the MCBS, the validity checks offered compelling evidence that the recommended threshold-identification method was effective in separating rapid-guessing behavior from solution behavior. A very low percent of rapid-guessing behavior was identified, as compared to existing results for different assessments. For this dataset, rapid-guessing behavior had minimum impact on parameter estimation in the IRT modeling. However, the students clearly exhibited different behaviors when they received items that did not match their performance level. We also found disagreement between students' response-time effort and self reports, but based on the observed data, it is unclear whether the disagreement was related to how the students interpreted the background questions. The paper provides a way to address the issue of identifying rapid-guessing behavior, and sheds light on the question about students' extent of engagement in NAEP and the impact, without relying on students' self evaluation or additional costs in test design. It reveals useful information about test-taking behaviors in a NAEP assessment setting that has not been available in the literature. The procedure is applicable to future standard NAEP assessments, as well as other tests, when timing data are available.

How Do Other Countries Measure Up to the Mathematics Achievement Levels on the National Assessment of Educational Progress?

In this study, we mapped achievement levels from the National Assessment of Educational Progress (NAEP) onto the score scales for selected assessments from the Trends in International Mathematics and Science Study (TIMSS) and the Program for International Student Achievement (PISA). The mapping was conducted on NAEP, TIMSS, and PISA Mathematics assessments in 2003. A focus of the study was on whether the NAEP achievement levels were set too high. The results indicated that students from many other countries had substantially larger percentages of students meeting NAEP mathematics achievement levels. In general, the findings suggest the NAEP standard for Advanced is high, but not too high when considered within an international context. With respect to the NAEP standard of Proficient, none of the top-performing countries approached 100% proficient, which seems to underscore the different conceptualizations of “Proficient” in NAEP and No Child Left Behind.

An Importance Sampling EM Algorithm for Latent Regression Models

Reporting methods used in large-scale assessments such as the National Assessment of Educational Progress (NAEP) rely on latent regression models. To fit the latent regression model using the maximum likelihood estimation technique, multivariate integrals must be evaluated. In the computer program MGROUP used by the Educational Testing Service for fitting the latent regression model to data from NAEP and other assessments, the integral is computed either by numerical quadrature or approximated. CGROUP, the current operational version of MGROUP used in NAEP for problems with more than two dimensions, uses Laplace approximation that may not provide fully satisfactory results, especially if the number of items per scale is small. This article examines a stochastic expectation-maximization (EM) method that uses importance sampling to NAEP-like settings. A simulation study and a real data analysis show that the importance sampling EM method provides a viable alternative to CGROUP for fitting multivariate laten...

Do National and State Assessments Converge for Educational Accountability? A Meta-Analytic Synthesis of Multiple Measures in Maine and Kentucky

Given the policy imperative of using multiple measures for state education accountability under the No Child Left Behind Act (NCLB), this study examines similarities and discrepancies between the National Assessment of Educational Progress (NAEP) and the states' own math assessment results in Kentucky and Maine, with a focus on 3 major academic performance indicators: proficiency level, achievement gap, and achievement gain. Using meta-analytic techniques, the study synthesizes multiple measures from the two states over the periods of 1992–1996 and 2000–2003. It pinpoints the areas and degrees of the discrepancies and explores contributing factors. It also reports emerging convergence of the NAEP and state assessments under the NCLB.

The 16th Bracey Report on the Condition of Public Education

NEWT GINGRICH has suggested that Democrats run their election campaigns on a simple slogan: Had enough? (1) Bush Administration shenanigans and Congressional political ploys so brazen as to be unbelievable--for instance, tying a rise in minimum wage that would benefit millions to a reduction in estate tax that would benefit nation's 7,500 wealthiest families--had me nodding in agreement. Herewith, year in review. DESPERATELY SEEKING STRAWS TO GRASP This law is helping us learn about what works in our schools. And clearly, high and accountability are working. Over last five years, our 9-year-olds have made more progress in reading than in previous 28 combined. So said Margaret Spellings at Child Left Behind Summit in April 2006, referring to gains on National Assessment of Educational Progress (NAEP). (2) In statistical circles, what secretary is doing is called picking. And in this instance, careless and self-serving cherry picking, too. Those last five years Spellings spoke of span 1999 to 2004. For two of those years Bill Clinton was President, and it is possible that all that gain--all 7 points--occurred on his watch. In two of those five years, NCLB did not exist. In 2001-02, NCLB (signed into law in January 2002) would have been in existence only three months before a NAEP assessment--had there been one. And given confusion that reigned from 2002 to 2004 and hostility between states and U.S. Department of Education, it is not likely that much gain occurred then. (Remember that in 2002, then-Secretary Rod Paige accused some states of trying to ratchet down their standards and thus of being enemies of equal justice and equal opportunity and ... apologists for failure. (3) It's surprising he didn't go on to call them terrorist organizations.) Spellings' statement is true only if you start in NAEP's first trend year, 1971. Begin in year of NAEP's previous trend high point, 1980, and gain would be only 4 points. Then, too, there was no gain from 1999 to 2004 for 13-year-olds and a decline of 3 points for 17-year-olds. And why didn't she mention math trends, since 9-year-olds showed a 9-point gain and 13-year-olds a 5-point gain? Seventeen-year-olds, though, showed a 1-point decline. Spellings also said, Scores are at all-time highs for African American and Hispanic students. (4) Well, if she meant reading scores for 9-year-olds, that was true. But it wasn't true for 17-year-old blacks or 13- and 17-year-old Hispanics (13-year-old blacks were at an all-time high by a single point). The statement would have been true, too, in mathematics, except for black and Hispanic 17-year-olds. The NAEP assessment of 2005, though, proved less upbeat. The regular NAEP assessments, ones billed as the nation's report card, change items over time in conjunction with curricular shifts; NAEP that yields trends administers same items at each assessment. In assessment for 2005, fourth-grade reading reached same level as it had at onset of NCLB in 2002, and eighth-grade reading declined 2 points. In math, scores rose 3 points for fourth-graders from 2003 and 1 point for eighth-graders. In reading, proportion of students at or above proficient level was static for fourth-graders at 31% and fell for eighth-graders from 33% to 31%. In mathematics, proportion of fourth-graders at or above proficient rose from 32% to 36%, while for eighth-graders it rose from 29% to 30%. While the Administration scrambled to put best face on numbers and to defend law that some complain forces a test-driven curriculum on classroom, Lois Romano reported in Washington Post, Ross Wiener of Education Trust had a more common reaction: No one can be satisfied with these results. There's been a discernible slowdown in progress since '03, at a time when we desperately need to accelerate gains. …

COGNITIVE DIAGNOSIS FOR NAEP PROFICIENCY DATA

ABSTRACTMore than a dozen statistical models have been developed for the purpose of cognitive diagnosis. These models are supposed to extract a much finer level of information from item responses than traditional unidimensional item response models. In this paper, a general diagnostic model (GDM) was used to analyze a set of simulated sparse data and real data from National Assessment of Educational Progress (NAEP) assessments. The purpose of this study was to find out (a) whether the parameters can be recovered for a sparse data matrix in the framework of the GDM and (b) how to estimate group characteristics for large survey data, such as NAEP, in the framework of the GDM. The results of the simulation study show that GDM parameters can be recovered satisfactorily. The GDM under single group and multiple group assumptions were employed to fit NAEP assessment data. The results under these two assumptions and between the GDM and the statistics derived using the operational NAEP model were compared. The findings indicate that it is possible to conduct cognitive diagnosis for NAEP proficiency data.

EXTENSION OF THE NAEP BGROUP PROGRAM TO HIGHER DIMENSIONS

The reporting methods used in large scale assessments such as the National Assessment of Educational Progress (NAEP) rely on a latent regression model. The first component of the model consists of a p-scale IRT measurement model that defines the response probabilities on a set of cognitive items in p scales depending on a p-dimensional latent trait variable θ = (θ1, … θp). In the second component, the conditional distribution of this latent trait variable θ is modeled by a multivariate, multiple linear regression on a set of predictor variables, which are usually based on student, school and teacher variables in assessments such as NAEP. To fit the latent regression model using the maximum (marginal) likelihood estimation technique, multivariate integrals have to be evaluated. In the computer program MGROUP used by ETS for fitting the latent regression model to data from NAEP and other programs, the integration is currently done either by numerical quadrature for problems up to two dimensions or by an approximation of the integral. CGROUP, the current operational version of the MGROUP program used in NAEP and other assessments since 1993, is based on Laplace approximation, which may not provide fully satisfactory results, especially if the number of items per scale is small (see, e.g., Thomas, 1993a, or von Davier & Sinharay, 2004). There is scope for improvement in the technique used. This paper extends the NAEP BGROUP program to higher dimensions. Two real data analyses, one with a medium-sized data set and another with a large data set, show that the extension promises to be useful for fitting the NAEP model.

Measuring Student Achievement in the Future Based on Lessons from the Past: The NAEP Arts Assessment

1997, the National Assessment of Educational Progress (NAEP), also known as the Nation's Report ':,l, assessed eighth-grade music students throughout the United States on their ability to perform, creae! and respond to music. The next assessment in music is scheduled for 2008. If this assessment is to pride a realistic and useful look at what students in the US know and can do in music, music teachIr need to get involved in the process now, as the NAEP is making plans. We need to look carefully at the 1997 NAEP assessments, determine what could make the next assessment more valuable to the music education community, and make our voices heard to those who can make decisions regarding the upcoming assessment. This article provides information on the design of the NAEP and offers ideas for improving the assessment to make it more useful to the music education community.

Checking up on Charters

CAN YOU remember when single piece of educational research led to full-page ad in the New York Times? Me neither. But that is what happened in August 2004. A chronicle follows. It would be kind to say that the U.S. Department of Education was dawdling in its analysis of the data on charter schools that were collected by the National Assessment of Educational Progress (NAEP). ED collected the data with the regular NAEP assessment in 2003 and placed the regular data on its website in the fall of 2003. American Federation of Teachers found the charter school data, though, and analyzed them. New York Times reported the AFT's findings in front-page story on 17 August 2004. Charter schools did not come off well when compared to regular public schools, even when the analysis controlled for family income and location. There were no ethnic differences between the student populations of the regular public schools and the charters, but the white/minority achievement gap was large in charters it was in regular public schools. To say that the appearance of this article caused the Right to go ballistic is to practice understatement. August 18 saw an op-ed in the Wall Street Journal titled Dog Eats AFT's Homework, by Harvard's William Howell, Paul Peterson, and Martin West. Secretary of Education Rod Paige inveigled the Times to do another article and allow him to defend charters. Newsday's editorial page said that the jury was still out on charters. op-ed page of the Chicago Tribune said that the AFT's findings were as new lava lamp, revelatory an old sock, and significant belch (thereby indicating need for remedial simile instruction). editorial in the Times claimed that the findings were bad news for No Child Left Behind because the low performance of charters meant that it made no sense for chronically failing public schools to convert to charter status. On August 19, the Rev. Floyd Flake defended charters in New York Times op-ed. (In its credit line, the Times failed to point out that Flake is the president of the Charter Schools Division of Edison Schools, Inc.) In his New York Post op-ed, Defaming Charters, Chester Finn, Jr., labeled the analysis a mischief-bearing grenade, hand delivered by the charter-hating American Federation of Teachers. (Historical note: the Washington Times once called Finn himself bomb-thrower.) Finn's piece overlooked the fact that, without longtime AFT President Albert Shanker's backing of charters, there probably wouldn't be any. Shanker later became disillusioned and saw charters frivolous and divisive. Also on August 19, Jay Greene, Manhattan Institute fellow, published an op-ed in the New York Sun; the Center for Education Reform's Jeanne Allen appeared on NPR's Tavis Smiley Show; and ED's Nina Shokraii Rees showed up on The News- Hour with Jim Lehrer. But about that ad. It appeared on August 25, its $125,000 cost borne by Allen and her Center for Education Reform. ad, signed by 31 professors, scolded the AFT for its analysis and the New York Times for reporting the story. ad said, in part, The study in question does not meet current professional research It then described the standards that the AFT and the Times had violated. There is nothing wrong with those standards except that they constitute massive exercise in hypocrisy, because many of the signatories violate them regularly. signatories guilty of such violations include Paul Peterson and his group at Harvard; Caroline Hoxby, whose work was sharply criticized by Laurence Mishel in American Prospect; Howard Fuller, propagandist whose work does not approach being research; and Jay Greene, all of whose seven extant working papers from the Manhattan Institute violate one or more of the listed standards. ad appeared few weeks later in Education Week, minus two signers who told me they didn't know what they were getting into. …

APPLICATION OF THE STOCHASTIC EM METHOD TO LATENT REGRESSION MODELS

ABSTRACTThe reporting methods used in large scale assessments such as the National Assessment of Educational Progress (NAEP) rely on a latent regression model. The first component of the model consists of a p‐scale IRT measurement model that defines the response probabilities on a set of cognitive items in p scales depending on a p‐dimensional latent trait variable θ = (θ1, … θp). In the second component, the conditional distribution of this latent trait variable θ is modeled by a multivariate, multiple regression on a set of predictor variables, which are usually based on student, school and teacher variables in assessments such as NAEP.In order to fit the latent regression model using the maximum (marginal) likelihood estimation technique, multivariate integrals have to be evaluated. In the computer program MGROUP used by ETS for fitting the latent regression model to data from NAEP and other sources, the integration is currently done either by numerical quadrature (for problems up to two dimensions) or by an approximation of the integral. CGROUP, the current operational version of the MGROUP program used in NAEP and other assessments since 1993, is based on Laplace approximation that may not provide fully satisfactory results, especially if the number of items per scale is small.This paper examines the application of stochastic expectation‐maximization (EM) methods (where an integral is approximated by an average over a random sample) to NAEP‐like settings. We present a comparison of CGROUP with a promising implementation of the stochastic EM algorithm that utilizes importance sampling. Simulation studies and real data analysis show that the stochastic EM method provides a viable alternative to CGROUP for fitting multivariate latent regression models.

ANALYSIS OF NAEP COMBINED NATIONAL AND STATE SAMPLES

ABSTRACTThis National Assessment of Educational Progress (NAEP) study analyzes the efficacy of combined national and state NAEP samples. The main goal of combining NAEP national and state samples is to provide an integrated way to implement NAEP assessments and to reduce the burden of state assessments by administrating one assessment. The study is based on combined samples of the 1998 and 2000 NAEP assessments.

NAEP and No Child Left Behind: Technical Challenges and Practical Solutions

The National Assessment of Educational Progress(NAEP) is the only nationally representative andcontinuing assessment of what U.S. students knowand can do in various subject areas. The No ChildLeft Behind Act of 2001 requires that all U.S. states,jurisdictions, and territories submit plans to theSecretary of Education that demonstrate that thestate has adopted challenging academic contentand student achievement standards. As part of thatplan, states and school districts that receive Title Ifunds must participate in NAEP assessments inreading and math at grades 4 and 8 every twoyears. The purpose of NAEP has always been tosurvey what students know and can do and to monitorchanges over time. Now, NAEP has a newrole, to act as a serious discussion tool in evaluatingresults of state assessments and in providing acommon base for comparison between states. Thisrole brings new challenges and opportunities forthe NAEP program.

NAEP and No Child Left Behind: Technical Challenges and Practical Solutions

The National Assessment of Educational Progress (NAEP) is the only nationally representative and continuing assessment of what U.S. students know and can do in various subject areas. The No Child Left Behind Act of 2001 requires that all U.S. states, jurisdictions, and territories submit plans to the Secretary of Education that demonstrate that the state has adopted challenging academic content and student achievement standards. As part of that plan, states and school districts that receive Title I funds must participate in NAEP assessments in reading and math at grades 4 and 8 every two years. The purpose of NAEP has always been to survey what students know and can do and to monitor changes over time. Now, NAEP has a new role, to act as a serious discussion tool in evaluating results of state assessments and in providing a common base for comparison between states. This role brings new challenges and opportunities for the NAEP program.

Setting Performance Standards for Professional Licensure and Certification

Credentialing programs were surveyed to ascertain the procedures that they use to set performance standards on multiple-choice and open-ended assessments. For multiple-choice assessments, these programs mostly employ variations on the Angoff (1971) standard-setting method. Procedures used with open-ended questions showed more divergence; some agencies use a question by question approach, whereas others utilize methods that consider the assessment results more holistically. Implications of these standard-setting practices from credentialing agencies to the National Assessment of Educational Progress (NAEP), including the consequences of the assessment on the individual candidate, the matrix sampling construction of NAEP assessments, the multiple cutpoints of the NAEP assessment program, and the types of validity evidence that are typically gathered to support the validity of the performance standard, are discussed. Generalizations of these standard-setting methods from the field of professional licensure and certification should be made with caution.