National Assessment Governing Board Research Articles

Advances and Missed Opportunities in the Development of the 2026 NAEP Reading Framework

The National Assessment of Educational Progress (NAEP) in Reading aims to measure reading comprehension for students in the United States and to monitor progress in our education system. NAEP Reading is developed based on an assessment framework document that is periodically revised to reflect the latest understandings about reading comprehension and its assessment. A key goal of the Visioning Panel (VP) and the Development Panel (DP) charged with updating the NAEP Reading Assessment Framework for 2026 was to lay the foundation for an assessment that made progress toward greater fairness, equity, and validity. In this essay, we discuss how the Framework development process unfolded and its results. We document the unusual ways the National Assessment Governing Board (“the Board”) shaped the development process. We provide evidence that a small group of Board members aimed to preserve the status quo in reading assessment by downplaying reliance on expertise and authoritative sources of research on reading, learning, and assessment and by removing attention to equity in NAEP Reading. We also discuss both successful (i.e., approved by the Board) and unsuccessful (i.e., rejected by the Board) recommendations for changes to the 2026 Framework that initially were proposed by the DP. We end by considering how, despite the efforts of the small group of Board members, we as a literacy field can improve the nature and impact of our large-scale reading assessments, and NAEP in particular.

Measuring Academic Readiness for College

Recent years have seen increased attention to the high school–college transition. Students are enrolling in college in record numbers, yet they are also taking increasing numbers of remedial courses. How to measure and report on academic readiness for college is an important policy issue receiving attention from the National Assessment Governing Board, Achieve, and several states. The focus of this article is how to create a measure of academic readiness for college, either by building and validating a new assessment or validating and repurposing an existing assessment. After first describing the disjuncture between high school and college and discussing definitions and indicators of academic readiness, the authors identify four strategies that might be used to create a readiness assessment. The pros and cons of each strategy are discussed.

Reliability and Validity of Bookmark‐Based Methods for Standard Setting: Comparisons to Angoff‐Based Methods in the National Assessment of Educational Progress

Historically, Angoff‐based methods were used to establish cut scores on the National Assessment of Educational Progress (NAEP). In 2005, the National Assessment Governing Board oversaw multiple studies aimed at evaluating the reliability and validity of Bookmark‐based methods via a comparison to Angoff‐based methods. As the Board considered adoption of Bookmark‐based methods, it considered several criteria, including reliability of the cut scores, validity of the cut scores as evidenced by comparability of results to those from Angoff, and procedural validity as evidenced by panelist understanding of the method tasks and instructions and confidence in the results. As a result of their review, a Bookmark‐based method was adopted for NAEP, and has been used since that time. This article goes beyond the Governing Board's initial evaluations to conduct a systematic review of 27 studies in NAEP research conducted over 15 years. This research is used to evaluate Bookmark‐based methods on key criteria originally considered by the Governing Board. Findings suggest that Bookmark‐based methods have comparable reliability, resulting cut scores, and panelist evaluations to Angoff. Given that Bookmark‐based methods are shorter in duration and less costly, Bookmark‐based methods may be preferable to Angoff for NAEP standard setting.

The Real-World Reading and Writing U.S. Children Need

The 2001 Progress in International Reading Literacy (PIRLS) study--a study of the reading achievement of mostly 10-year-old students in 35 nations--found that U.S. students had the largest gap between literary reading achievement and informational reading achievement of any nation studied (Mullis et al. 2003), with the gap favoring literary reading. In the 2006 study, this gap narrowed somewhat, but average scores of U.S. students were statistically unchanged (Mullis et al. 2007). What causes this? In this age, why are U.S. students relatively poor at reading to acquire and use real-world information (informational texts) when compared to reading to experience imagined worlds (literary texts)? Some might say that informational reading is inherently more difficult than literary reading, but many nations had comparable literary and informational reading achievement, and some even had stronger informational achievement, so that explanation seems insufficient. Others might say that it's the nature of the particular reading assessment used in the PIRLS study, but studies using other assessments have also shown that U.S. students struggle with comprehension of informational text as compared to narrative or literary reading. A more plausible explanation is that U.S. students have little experience with informational text in the early years of schooling. In fact, they get very little. In a study I conducted in U.S. 1st-grade classrooms (Duke 2000), I found that only 9.8% of the books and other materials in classroom libraries and only 2.6% of the materials on classroom walls and other surfaces were informational text. More important, important, only 3.6 minutes of the classroom day, on average, were devoted to informational text--only 1.9 minutes in classrooms in low-socioeconomic-status settings. Studies that have looked at this issue in other ways--for example, by examining basal reading programs--have also found a relative lack of informational text (e.g., Moss and Newton 2002). When I tell others about this finding, a common response is that the focus of primary grade education should be on developing decoding and fluency, not comprehension and certainly not comprehension of informational text. However, we found that 1st graders whose teachers provided greater experience with informational text had decoding and spelling skills comparable to those of other students (Duke et al. 2009). Another response is that informational text is simply too hard, perhaps even developmentally inappropriate, for young children. This stance has long been noted, and countered, in research (Duke and Kays 1998; Newkirk 1987; Pappas 1993). Young children can be successful with informational text--in fact, some studies in this area (Maduram 2000; Tower 2002) have included three-and four-year-old children! There are some indications that attention to informational text in early schooling is increasing. For example, in 2008, Moss found more informational text in basal reading programs than in the past. This is encouraging-informational reading ability is arguably more important now than it ever has been. Reflecting this, the 2009 National Assessment of Educational Progress (NAEP) Framework calls for 50% informational passages by 4th grade, 55% by 8th grade, and 70% by 12th grade (National Assessment Governing Board 2007). In sum, there is little question about whether we should involve informational text in early schooling; the question now is how. THE USUAL SUSPECTS So far, most instructional practices shown to help develop young children's comprehension of informational text come as no surprise. For example, just as teaching young children the structure of stories has long been shown to improve their comprehension of stories, so, too, does teaching young children the structures of informational text (specifically, compare-contrast and cause-effect) improve their comprehension of these texts (Williams 2005, with 2nd graders). …

Evaluation of the National Assessment of Educational Progress: Next Steps

The report, Evaluation of the National Assessment of Educational Progress, provides a number of recommendations for addressing validity concerns about NAEP. This article identifies actions that could be taken by the Congress, the National Center for Education Statistics, and the National Assessment Governing Board— which share responsibility for NAEP's validity— in responding to those concerns.

Evaluation of the Standard Setting on the 2005 Grade 12 National Assessment of Educational Progress Mathematics Test

The National Assessment Governing Board used a new method to set achievement level standards on the 2005 Grade 12 NAEP Math test. In this article, we summarize our independent evaluation of the process used to set these standards. The evaluation data included observations of the standard-setting meeting, observations of advisory committee meetings where the results were discussed, review of documentation associated with the standard-setting study, analysis of the standard-setting data, and analysis of other data related to the mathematics proficiency of 2005 Grade 12 students. Our evaluation framework used criteria from the Standards for Educational and Psychological Testing (AERA, APA, & NCME 1999) and other suggestions from the literature (e.g., Kane, 1994, 2001). The process was found to have adequate procedural and internal evidence of validity. Using external data to evaluate the standards provided more equivocal results. In considering all evidence and data reviewed, we concluded the process used to set achievement level standards on the 2005 Grade 12 NAEP Math test was sound and the standards set are valid for the purpose of reporting achievement level results on this test. Recommendations for future NAEP standard-setting studies are provided.

A Study of Core-Plus Students Attending Michigan State University

1. INTRODUCTION. One important measure of the effectiveness of a high school mathematics program is the success students have in subsequent university mathematics courses. Yet this measure is seldom considered in studies of high school curricula. This article describes a study that we undertook to quantify this measure for one particular curriculum. Readers of this MONTHLY may be aware of the general context leading to the need for such evaluative studies. Over the past two decades there has been a growing awareness of the inadequacy of the mathematical skills of American high school graduates. That was the assessment of the 1983 report A Nation at Risk [14] and confirmed by many subsequent studies. A recent National Assessment of Educational Progress (NAEP) Report [3] concluded that only 17 percent of U.S. twelfth graders were “proficient” at mathematics. 1 International comparisons also indicate a relatively low level of mathematics achievement by U.S. high school students. The Third International Mathematics and Science Study (TIMSS) assessed the “Mathematics Literacy” of end-ofsecondary students in twenty-two countries and found that U.S. students outperformed only two countries, Cyprus and South Africa [18]. Related studies suggest that the mathematics courses taken by American high school students are often at a lower level than those taken by their international peers and that U.S. high schools are offering a wide assortment of courses that lack the focus and coherence found in many foreign curricula [19]. This situation has been of particular concern on college and university campuses, where large numbers of entering students require remedial courses to bring their mathematical knowledge and skills up to what is required for a wide variety of college courses. One effort to improve school mathematics began in 1989 with the publication of Curriculum and Evaluation Standards for School Mathematics (generally known as “The NCTM Standards”) by the National Council of Teachers of Mathematics [5]. The National Science Foundation (NSF) subsequently funded the development and implementation of thirteen elementary, middle, and high school mathematics curricula based on theseStandards, programs variously referred to as “reform,” “Standardsbased,” or “NSF-sponsored” by their publishers and others. Many of these programs have been controversial. In particular, there has been concern that the NSF-sponsored curricula moved from pilot testing to large-scale implementation without sufficient independent evaluations of their efficacy in preparing students for college mathematics and science courses. Although studies of the effectiveness of these curricula have yielded promising conclusions (see, for example, the collection of such evaluations in [17]), most have been conducted by persons associated with their writing or implementation. A recent National Research Council (NRC) report, On Evaluating Curricular Effectiveness [8], also expresses concerns about the methodological adequacy of these studies. This NRC 1 The notion of “proficient” used in this statistic is identified by the National Assessment Governing Board

Issues and Options in Creating a National Assessment in World History

THE NATIONAL ASSESSMENT Governing Board (NAGB) is considering creating a National Assessment of Educational Progress (NAEP) for world history education.' On the surface, a national assessment in world history appears to be a sensible and essentially unproblematic decision. NAGB already has assessments in United States History, Geography, Civics, and Economics (to be tested for the first time in 2006). As a school subject, world history in one form or another appears to be the fastest growing segment of the American school curriculum. Over the past twenty years almost every state has added world history related content to its curriculum at some grade level and in some form. Many require a course in world history for high school graduation. Perhaps the most dramatic indicator of world history's popularity has been the development and growth of the College Board's Advanced Placement (AP) World History course. Tested for the first time in the spring of 2002, the AP World History exam drew the largest first time subscription in

Editorial

Editorial Testing......3,2,1 Recently I read some material about the conflicting and confusing results of national and international testing efforts in general education. There seem to be two possible ways of interpreting the data: One, the United States is among world leaders in developing reading, math, and science skills in our school children; or Two, we are doing an abysmal job, and our children lack basic, reading, math, and science skills. Because of this, our children will be doomed to careers of weed pulling, broom pushing, and hamburger flipping, punctuated by unemployment and poverty. There is, of course, a third possibility: Perhaps all countries somehow are producing mindless troglodytes , unlike we highly educated and intelligent adults. I hope the (adult) readers of this editorial will not take the third possibility seriously, but I do want to address the first two issues and then try to relate them to testing of deaf children. The question of valid and reliable testing is assuming greater importance in the United States. At the time of preparation of this editorial, it appears that congress will mandate annual testing of children from grades three to eight, and possibly more. There will probably be compromises between state control of testing and federal requirements , so that there will continue to be some variation within a general trend toward uniformity in testing. From my limited observations, several states already have developed reliable and valid procedures, and several have not. I would expect this to continue, but with some overall improvement . At present there are literally thousands of tests being employed . The assumptions behind the tests and the goals vary, with a predictable variation in results. To a great extent, the results reported represent different orientations of norm-referenced and criterion-referenced tests. For example, tests that compare children from different countries , states, cities or schools do not necessarily provide information about content or about excellence. It is within this context that so much confusion exists. The above-mentioned relatively high reading achievement scores for American children come from the International Association for the Evaluation of Educational Achievement. The math and science results come from the Third International Math and Science Study. The results, per se, tell us nothing about achievement. The poor results come from the National Assessment of Educational Progress (NAEP), which sets four levels in its testing for different grades: advanced, proficient, basic, and below basic. The tests are supervised by the congressionally appointed National Assessment Governing Board. For one year, almost 40% of American fourth graders were rated below basic level in reading, 36% in math, and 33% in science. The results appear grim. Which sets of tests are more valid? It is difficult to say. Certainly they are different. The international math, science , and reading fourth grade tests ask children to perform basic tasks, for example addition, subtraction, multiplication and division in math. The NAEP adds word problems. On the surface, it might appear that the competency -based NAEP tests are preferable. However, there is no consensus on this. Some critics argue that the NAEP grading scale is unrealistically strict and underestimates achievement. Supporters of the NAEP tests say these are hard and stiffly graded, but that this is positive because the high standards will help schools improve. Volume 146, No. 3, 2001 American Annals of the Deaf Editorial The arguments for different approaches to testing are complex and will not be resolved in the near future. Our immediate concern is that deaf children will participate in the various testing programs, and we have little or no understanding of the implications of this testing. If there continues to be variation across states, the possibility exists that deaf children will "fall between the cracks." Many may be denied promotion or graduation because of inappropriate testing. We have had no nationwide achievement data on deaf school-age children for five years now, and prospects for the future are uncertain. Improved testing could provide an opportunity to develop approaches to benefit deaf children that could be incorporated into individual academic education plans, but this will happen only with a major national effort that transcends states boundaries. It would seem that this would be a logical area...

Regional Findings from a Secondary Analysis of the 1997 NAEP Art Assessment Based on Responses to Creating and Responding Exercises

In 1969 Congress charged U.S. Department of Education through National Center for Educational Statistics (NCES) with developing and implementing nationally representative and continuing measurement of student achievement in American schools. Assessment of visual arts achievement began with 1974 National Assessment of Educational Progress (NAEP) and continued in 1978. Findings from these assessments provided initiative for similar high stakes, large-scale, state level assessments of visual arts achievement in 1980s (Sabol, 1990, 1994, 1998a; Shuler & Connealy, 1998). However, subsequent NAEP visual arts assessments were not forthcoming. It was not until 17 years later that visual arts were included again as part of NAEP. Only after arts was added to nation's goals as a core subject (U.S. Department of Education, 1994) and Congress stated that the arts are forms of understanding and knowing that are fundamentally important to education (U.S. Congress, 1994, p 2) were arts reintroduced in assessment. Subsequently, assessments were done in visual arts, music, and theater. Dance was not included in 1997 NAEP. Each student who participated in assessment was assessed in one of three arts. The National Assessment Governing Board (NAGB) was established to formulate policy guidelines for NAEP. It is responsible for selecting subject areas to be assessed, developing assessment objectives, identifying appropriate achievement goals for each grade level and subject tested, and establishing standards and procedures for interstate and national comparisons (National Assessment Governing Board, 1997). The NAGB divided processes involved in visual arts into creating and responding categories and outlined specific knowledge and skills content for each category in its assessment framework for arts (National Assessment Governing Board, 1997). The NAEP framework was deliberately designed to reflect best practices from field and to be compatible with national arts standards (Music Educators National Conference, 1994; National Assessment Governing Board, 1997). Assessment exercises included authentic tasks that assessed knowledge and skills in creating and responding to visual arts. Independent sets of exercises on 1997 NAEP in art were divided into Creating and Responding blocks. Items in these blocks focused on content knowledge, skills, and processes used in making and responding to art. Measurement of students' Creating and Responding ability also is compatible with national visual arts standards (Music Educators National Conference, 1994), visual arts curriculum development (Sabol, 1994), and visual arts classroom practices. A total of eight blocks were developed including three Creating blocks, two Responding blocks, and three blocks that combined assessment of creating and responding knowledge and skills in visual arts.1 No student completed all blocks of assessment. Students completed either one Responding block and one Creating block or two Responding blocks. In order to measure students' knowledge and skills in creating, students were required to create and work with various media to make visual art. Students showed abilities to communicate in works of art, think of different solutions to visual problems, generate for works of art and design, and use their technical proficiency in realizing those ideas (Persky, Sandene, & Askew, 1998, p. 80). They also were asked to evaluate their own work in written form. Evaluation questions asked students to identify, describe, and analyze important qualities in their work. Responding blocks included exercises in which students viewed works of art representing a variety of cultures, genres, and historical periods. Responding questions focused on students' ability to demonstrate their knowledge of media and techniques, visual organization, cultural contexts of artworks, and how user needs are met with well-- designed functional objects. …

The Controversy over the National Assessment Governing Board Standards

This paper provides an analysis of the controversy surrounding the standard setting process conducted by ACT Inc. for the National Assessment Governing Board (NAGB). 1 This process is the most thoroughly planned, carefully executed, exhaustively evaluated, completely documented, and most visible of any standard setting process of which I am aware. Extensive research was conducted to determine how best to develop each step in the process. 2 A distinguished team of experts guided the process through its development and implementation. 3 And, the process has been open to scrutiny with evaluators observing the design and implementation of every step.

Comment

Mark D. Reckase has been closely involved in the achievement level setting (ALS) process that ACT Inc. has carried out under contract for the National Assessment Governing Board (NAGB). In his paper, he offers an overview of that process and argues that both the process and the resulting standards are sound. He then discusses interpretation of standards-based National Assessment of Educational Progress (NAEP) reports in the light of motivational context and test-curriculum match. Next, Reckase summarizes [End Page 255] criticisms of the standards, suggests reasons for those criticisms, and offers a rebuttal. He concludes that the process is "the most carefully developed, thoroughly researched, and frequently evaluated standard setting process that has been conducted to date."

Lessons in Contracting for Evaluations

A 1995 evaluation of the U.S. Marine Corps’ personnel evaluation system and a 1991 evaluation of the National Assessment Governing Board procedure for setting cut scores on the mathematics section of the National Assessment of Educational Progress provided valuable lessons in how to minimize the risks of misunderstanding what an evaluation will involve, subversion of the evaluation, controversy, and animosity. Both evaluations were nationally significant, had to be conducted quickly, were politically volatile, were keyed to professional standards, and had substantial impacts. However, the latter evaluation went sour, whereas the former received an official commendation. Evaluators should do all they can to demonstrate to clients that sound evaluations are not to be feared, but should be valued and used. An investigation of what went right and wrong in these evaluations identified contracting as a key variable. This article advises evaluators and clients to regularly negotiate clear, sound contracts before proceeding with an evaluation, and presents a checklist to assist in the contracting process.

Conceptions of Science Achievement in Major Reform Documents

The construct of science achievement—what K-12 students should know and be able to do in science—is central to science education reform. This paper examines current conceptions of science achievement in major reform documents in the context of standards-based and systemic reform. The paper reviews documents on (a) science content standards, including the National Science Education Standards (National Research Council, 1996) and Project 2061 (American Association for the Advancement of Science, 1989, 1993); (b) performance standards in the New Standards Project (National Center on Education and the Economy, 1997a, 1997b, 1997c, 1998); and (c) assessment frameworks, including the 1996 National Assessment of Educational Progress (National Assessment Governing Board, 1994, 1996) and the Third International Mathematics and Science Study (Martin & Kelly, 1996; McKnight, Schmidt, & Raizen, 1993; Robitallie et al., 1993). Although there is an overall agreement on the conceptions of science achievement among the documents, there are also noticeable differences. Based on the analysis of the five sets of documents, an aggregated view of science achievement is presented in terms of science content and process. Implications for promoting science achievement in standards-based and systemic reform are discussed.

Lessons in contracting for evaluations

A 1995 evaluation of the U.S. Marine Corps’ personnel evaluation system and a 1991 evaluation of the National Assessment Governing Board procedure for setting cut scores on the mathematics section of the National Assessment of Educational Progress provided valuable lessons in how to minimize the risks of misunderstanding what an evaluation will involve, subversion of the evaluation, controversy, and animosity. Both evaluations were nationally significant, had to be conducted quickly, were politically volatile, were keyed to professional standards, and had substantial impacts. However, the latter evaluation went sour, whereas the former received an official commendation. Evaluators should do all they can to demonstrate to clients that sound evaluations are not to be feared, but should be valued and used. An investigation of what went right and wrong in these evaluations identified contracting as a key variable. This article advises evaluators and clients to regularly negotiate clear, sound contracts before proceeding with an evaluation, and presents a checklist to assist in the contracting process.

Converting Boundaries Between National Assessment Governing Board Performance Categories to Points on the National Assessment of Educational Progress Score Scale: The 1996 Science NAEP Process

National Assessment Governing Board (NAGB) policy indicates that results from the National Assessment of Educational Progress (NAEP) should be reported according to the percentage of students estimated to be above 3 levels of standards called achievement levels. The standards, labeled Basic, Proficient, and Advanced, are operationalized by 3 points on the NAEP scale. In this article, I provide an overview of the process that was used to identify provisional locations for the points that would inform NAGB as they set the achievement levels for the science NAEP. The process includes the identification of panelists to be involved in the achievement-level setting, the training for the panelists, and the method for converting panelists ratings of NAEP items to points on the NAEP score scale.

Describing Performance Standards: Validity of the 1992 National Assessment of Educational Progress Achievement Level Descriptors as Characterizations of Mathematics Performance

For its first 2 decades, the National Asessement of Educational Progress (NAEP) provided only descriptive information on the academic of American youth. The 1988 reauthorization of NAEP, however, established the National Assessment Governing Board (NAGB) and gave it responsibility for establishing appropriate goals for each age and grade level in each subject area. The response of the NAGB, which has been a model for standards-based reporting in some state assessments, was to establish 3 achievement levels with a set of verbal descriptions and exemplar items. Reported in this article is a series of studies undertaken to evaluate the validity of the descriptions and exemplars as characterizations of the actual mathematics performance of students at the levels. Found in the study, were serious inconsistencies between actual performance and both the descriptions and exemplars provided by the NAGB. This article offers a number of recommendations in response, some of which apply to standards-bas...