State Achievement Tests Research Articles

Growth on 2019 State Achievement Tests: Empirical Benchmarks and the Role of Scale Choice

Benchmarks for “years of learning”–expectations for annual academic growth in standard deviation units–are popular to contextualize the practical significance of a causal effect. Though they are applied in studies using a variety of outcome measures, these benchmarks must be built using vertically scaled test scores. Vertical scales must be established to measure growth across grades, and different scales can lead to different conclusions about how much students grow. This article introduces new benchmarks for annual growth based on 2019 grade 3–8 state achievement tests in math and ELA, which are more appropriate for use in contemporary studies using test scores as an outcome measure than existing benchmarks based on older tests aligned to prior generation standards. While high-level trends in growth are generally consistent between prior studies and the present study, differences do arise that would lead to substantively different conclusions about the practical significance of a causal effect. Based on exploratory analyses of growth differences across 2019 achievement scales, further guidance is provided on when and how to use this type of aggregated benchmark for annual growth.

Using Content Relevance and Representativeness Indices in Instrument Revision

ABSTRACT Combining methods from earlier content validity and more contemporary content alignment studies may allow a more complete evaluation of the meaning of test scores than if either set of methods is used on its own. This article distinguishes item relevance indices in the content validity literature from test representativeness indices in the subsequent content alignment literature. We then demonstrate that setting minimum thresholds for both types of indices in a sequential process can affect conclusions about test content representativeness, which are integral to validity evaluation. We illustrate this outcome using test item-to-target domain objective match data from subject-matter expert panelists who reviewed sixteen US state achievement tests. Although the illustration incorporates educational achievement test data, the process has broader applicability to development of survey questionnaires, workplace tests, and other instruments.

Long-Term Impacts of Reading Recovery through 3rd and 4th Grade: A Regression Discontinuity Study

To estimate the long-term effects of the Reading Recovery® intervention, a regression discontinuity design (RD) was implemented in a randomly selected sample of Reading Recovery schools during each year of the federally-funded i3 Scale-Up external evaluation (2011–2015) and also in one additional cohort during the 2016–2017 school year. Long-term outcomes were measured by collecting scale scores on state achievement tests in reading or English language arts (ELA) that were administered to students during 3rd and 4th grades (2013–2019). We were able to record and link 3rd grade state test score data in reading/ELA for 9,879 students, and 4th grade state test data for 6,345 students. Results suggest that the long-term impact of Reading Recovery on students’ reading/ELA test scores in 3rd and 4th grades is statistically significant and substantially negative; students who participated in Reading Recovery in first grade had 3rd and 4th grade state test scores in reading/ELA that were, on average, .19 to .43 standard deviations (about one-half to one full grade level) below the state test scores of similar students who did not participate in Reading Recovery.

Current nationwide practices in U.S. Test accommodations for active bilingual learners/users of English (ABLE) students

ABSTRACT Schools are encountering a unique problem regarding how to best accommodate the increasing number of active bilingual learners/users of English (ABLE) students on state achievement tests. Previously, states used disability focused rather than language responsive accommodations for these students. The current study employed double-coding procedures to determine the number of tools, supports, and accommodations allowed for ABLE students across all U.S. states. Results revealed substantial heterogeneity of state practices, as some states demonstrated a decrease in the number of accommodations over time while some states offered more language responsive accommodations. Future research should continue to evaluate the appropriateness of the accommodations for ABLE populations.

The Long-Term Effects of Universal Preschool in Boston

AbstractWe use admissions lotteries to estimate the effects of large-scale public preschool in Boston on college-going, college preparation, standardized test scores, and behavioral outcomes. Preschool enrollment boosts college attendance as well as SAT test taking and high school graduation. Preschool also decreases high school disciplinary measures including juvenile incarceration, but has no detectable effect on state achievement test scores. An analysis of subgroups shows that effects on college enrollment, SAT-taking, and disciplinary outcomes are larger for boys than for girls. Our findings illustrate possibilities for large-scale modern, public preschool and highlight the importance of measuring long-term and non–test score outcomes in evaluating the effectiveness of education programs.

"Effects of a statewide pre-kindergarten program on children's achievement and behavior through sixth grade": Correction to Durkin et al. (2022).

Reports an error in "Effects of a statewide pre-kindergarten program on children's achievement and behavior through sixth grade" by Kelley Durkin, Mark W. Lipsey, Dale C. Farran and Sarah E. Wiesen (Developmental Psychology, 2022[Mar], Vol 58[3], 470-484). In the supplemental materials for the article, there was an error in Tables S1 and S3. An incorrect reverse-coded version of the treatment condition variable was used in the logistic regression models, causing the signs of the coefficients to be reversed, although the magnitude and p values were correct. The reported coefficients and odds ratios have been updated to have the correct direction of effects. (The following abstract of the original article appeared in record 2022-18712-001.) As state-funded pre-kindergarten (pre-K) programs expand, it is critical to investigate their short- and long-term effects. This article presents the results through sixth grade of a longitudinal randomized control study of the effects of a scaled-up, state-supported pre-K program. The analytic sample includes 2,990 children from low-income families who applied to oversubscribed pre-K program sites across the state and were randomly assigned to offers of admission or a wait list control. Data through sixth grade from state education records showed that the children randomly assigned to attend pre-K had lower state achievement test scores in third through sixth grades than control children, with the strongest negative effects in sixth grade. A negative effect was also found for disciplinary infractions, attendance, and receipt of special education services, with null effects on retention. The implications of these findings for pre-K policies and practices are discussed. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

A cost-effectiveness analysis of four approaches to universal screening for reading risk in upper elementary and middle school

Research on universal screening in reading has primarily focused on the psychometric adequacy of screening procedures without critically considering costs and value. Educators in upper elementary and middle school have access to a great deal of extant student achievement data, which makes the evaluation of the costs associated with collecting new data for screening purposes paramount. We conducted a retrospective analysis of four approaches to reading screening using cost-effectiveness analysis. Universal screening (i.e., aimswebPlus reading) and statewide reading test data were collected in a midsize school district in Texas for 19,417 students in Grades 4–8. We analyzed the total cost, the classification accuracy based on local cut-scores derived using receiver operating characteristic analysis, and the cost-effectiveness of each approach. Full implementation of the fall administration of aimswebPlus reading cost the district $55,199 in upper elementary and $76,832 in middle school. Both the use of prior-year state achievement test data alone (which represented no additional costs for screening) and the multivariate model (i.e., prior-year state achievement test data plus aimswebPlus) met typical recommendations for classification accuracy. A multiple-gate model that used prior-year state test data and fall aimswebPlus reading on a low-achieving subset of students produced the lowest cost-effectiveness ratio at $156 per additional student accurately classified in upper elementary and $181 in middle school, but this model also led to the highest number of false negatives. The use of aimswebPlus alone resulted in the highest costs and lowest classification accuracy. The results suggest that the use of prior-year statewide achievement test data alone in Grades 4–8 is an efficient approach to universal screening for reading risk that may allow schools to shift resources from screening to other educational priorities.

Evidence-Based Reading Instruction for Secondary Students With Reading Difficulties Within Multitiered Systems of Support.

Christian Scott recently returned home from college and was set to begin his first year as a middle school special education teacher. During his first meeting with his new principal, he learned about his teaching schedule. His principal, Mrs. Walker, explained their district was making a push for using multitiered systems of support (MTSS) as a framework for instruction. Her school would identify students with reading difficulties based on their performance on prior state reading tests and use this information to develop student schedules that allow students to receive additional reading interventions (i.e., Tier 2, Tier 3). Christian would teach reading to students with disabilities on his caseload across instructional tiers (Tiers 1-3). He would be tasked with providing co-teaching support for students with disabilities as they participate in general education classes (i.e., Tier 1). He would also provide additional Tier 2 supports to students with and without disabilities that did not pass the state achievement test in reading. Finally, he would provide intensive interventions to students with disabilities who required small group instruction (Tier 3). Having just received his special education teaching credential, Christian was familiar with the co-teaching service delivery model and the elements of effective instruction for middle school students with reading difficulties. However, he wondered, "What will this actually look like? How will I support the general education teacher providing Tier 1 supports? It sounds like I will also provide Tier 2-type instruction to students via a reading intervention class. What should this look like? How will this differ from the small group, Tier 3 instruction I need to provide?" Christian's mind raced with questions he was too nervous to ask in his first meeting with his new supervisor. The special education lead teacher noticed Christian seemed unsure how to respond. She jumped in, "Don't worry, Christian. School doesn't start for a few weeks. I'll help you with the details so you can hit the ground running." Christian felt excited about the challenge but also overwhelmed. Planning lessons for students with varying needs across instructional tiers was a tall order, and he felt unsure about how to get started.

Effects of a statewide pre-kindergarten program on children's achievement and behavior through sixth grade.

As state-funded pre-kindergarten (pre-K) programs expand, it is critical to investigate their short- and long-term effects. This article presents the results through sixth grade of a longitudinal randomized control study of the effects of a scaled-up, state-supported pre-K program. The analytic sample includes 2,990 children from low-income families who applied to oversubscribed pre-K program sites across the state and were randomly assigned to offers of admission or a wait list control. Data through sixth grade from state education records showed that the children randomly assigned to attend pre-K had lower state achievement test scores in third through sixth grades than control children, with the strongest negative effects in sixth grade. A negative effect was also found for disciplinary infractions, attendance, and receipt of special education services, with null effects on retention. The implications of these findings for pre-K policies and practices are discussed. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Addressing the Need for Spanish Literacy Assessments Within the Context of Bilingual MTSS: Investigating the Technical Adequacy of ISIP Español for Grades 3–5

Building on theories of cross linguistic transfer, reading proficiency of Spanish-speaking Emerging Bilinguals may be enhanced by providing tiered instruction through multitiered systems of support. To realize the benefits of this systems-level framework, technically adequate universal screening assessments to support teachers’ decision making are needed to provide educators with information about students’ English and Spanish literacy skills. The purpose of this study is to investigate the technical adequacy of the ISIP Español universal screening assessment with Spanish and English outcome measures for Grades 3–5. Results suggested moderate concurrent and predictive validity evidence for the criterion assessments in both languages while classification accuracy results indicate comparable predictive utility of the Spanish screener for English and Spanish versions of the state achievement test but better predictive utility for a standardized achievement test of Spanish reading. Practical implications and directions for future research are discussed. Impact Statement Monitoring the literacy development of Spanish-speaking Emerging Bilinguals (EBs) in the context of bilingual programs should include assessments of Spanish literacy. However, there are few universal screening assessments of Spanish literacy for use within the context of Multitiered Systems of Support. This study aims to address this gap by exploring the validity and classification accuracy of the ISIP Español universal screening assessment of Spanish literacy for English and Spanish outcome measures. Supplemental data for this article can be accessed online at https://doi.org/10.1080/2372966X.2021.1986357.

Professional Learning Plan: A Tool to Facilitate Teacher Collaboration and Improve Instructional Practice

This case study is designed to support the preparation of school administrators and teacher leaders enrolled in courses pertaining to school improvement, instructional leadership, and educational change. The principal of an urban middle school has been charged by the district superintendent to study a downward trend in math state achievement test scores. The case involves a middle school principal and how he might choose to work with his faculty to analyze the issue and develop an appropriate professional learning plan to improve instructional practice. This case scaffolds leader preparation candidates in (a) collaboratively identifying a problem underlying declining state achievement test scores and (b) developing a professional learning plan to support school improvement efforts.

State Assessment Score Reporting Practices for English Learner Parents

AbstractThis study sought to investigate how states communicate results for academic achievement and English language proficiency (ELP) assessments to parents who are English learners (EL). This objective was addressed by evaluating: (a) whether score reports and interpretive guides for state academic achievement and ELP assessments in each state were translated for EL parents; and (b) if so, whether recommended score reporting guidelines were followed in practice. Results demonstrated that for state achievement tests, 29 states had translated score reports and 28 had translated interpretive guides. Nearly every state translated these materials for their ELP assessments in a wide variety of languages. Across ELP and state achievement assessments, most states were found to limit statistical jargon, utilize figures/graphics to communicate test results, and include follow‐up information for parents, which represent improvements observed in prior reviews. However, states rarely provided personalization, statements on intended score use, a student's score history, or a direct link to their interpretive guide in their score reports. Recommendations on making score reports and interpretive guides more accessible and interpretable for EL parents are discussed.

Evaluating the Impact of Rater Effects on Behavior Rating Scale Score Validity and Utility

Behavior rating scales represent one of the most commonly used types of assessments in school psychology. Yet, they suffer from a fundamental limitation: They are an indirect methodology influenced partially by student behavior and partially by rater perspectives. Thus, the current study utilized advanced analytic approaches to evaluate rater effects on the Academic Competence Evaluation Scales–Short Form–Teacher (ACES-SF-T) with a partially crossed sample of 132 fourth- and fifth-grade students rated by seven teachers. Results indicated that rater effects had a minimal impact on the predictive validity of ACES-SF-T scores for state achievement tests, but at the individual level, rater effects could lead to starkly different conclusions about students’ academic, social, and behavioral functioning. Implications for research and practice are discussed.

Longitudinal Associations Linking Elementary and Middle School Contexts with Student Aggression in Early Adolescence.

Growing up in poverty increases youth risk for developing aggressive behavior problems, which, in turn, are associated with a host of problematic outcomes, including school drop-out, substance use, mental health problems, and delinquency. In part, this may be due to exposure to adverse school contexts that create socialization influences supporting aggression. In the current study, 356 children from low-income families (58% White, 17% Latinx, 25% Black; 54% girls) were followed from preschool through seventh grade. Longitudinal data included measures of the school-level contexts experienced by study participants during their elementary and middle school years, including school levels of poverty (percentage of students receiving free or reduced-price lunch) and academic achievement (percentage of students scoring below the basic proficiency level on state achievement tests). Regression analyses suggested little impact of these school-level contexts on teacher or parent ratings of aggression in fifth grade, controlling for child baseline aggression and demographics. In contrast, school-level contexts had significant effects on child aggression in seventh grade with unique contributions by school-level achievement, controlling for child fifth grade aggression and elementary school contexts along with baseline covariates. These effects were robust across teacher and parent ratings. Findings are discussed in terms of understanding the school-based socialization of aggressive behavior and implications for educational policy and prevention programming.

The Importance of Growth in Oral Reading Fluency to Predict Performance on High-Stakes Assessments among Students Receiving Supplemental Intervention

Educators and researchers measure student improvement in words read correct per minute (WRCM) to evaluate student response to reading interventions. The utility of evaluating improvement in WRCM via curriculum-based measures to make instructional decisions is based upon the assumption that growth in WRCM is predictive of performance on meaningful distal outcomes (e.g., state achievement tests). This study explored the relative value of measuring growth in WRCM to predict performance on an end of year state achievement test after controlling for baseline performance. We used quantile regression to analyze outcomes from 449 grade three students receiving tier II reading interventions to measure the relationship between growth in WRCM and performance on an end of year test across different levels of performance on that test. Results suggest that growth in WRCM is a meaningful predictor of performance on the end of year test after statistically controlling for baseline performance amongst students that perform better on that test (> 50th percentile). However, the amount of explained variance was consistently small (< 5%). Progress monitoring measures that more closely resemble the criterion of interest may be worth exploring for students that have made the transition from fluency to comprehension, while measures targeting lower order skills may be warranted for students with persistent fluency deficits.

Do Student-Level Incentives Increase Student Achievement? A Review of the Effect of Monetary Incentives on Test Performance

Background Policymakers have debated whether test scores represent students’ maximum level of effort, prompting research into whether student-level financial incentives can improve test scores. If cash incentives are shown to improve students’ test performance, there can be questions as to whether test scores obtained in the absence of financial incentives accurately reflect what students know and can do. This can raise concerns as to whether test scores should be used to guide policy decisions. Purpose This study used meta-analysis to estimate the effect of student-level incentives on test performance. The study also included a narrative review. Research Design: Twenty-one studies conducted in the United States and internationally were included in the meta-analysis. Effect sizes were estimated separately for mathematics, reading/language arts, and overall achievement. Findings Financial incentives had a significantly positive effect on overall achievement and on mathematics achievement, but no effect on reading/language arts achievement. The narrative review suggested mixed effects with regards to whether treatment estimates could be sustained after the removal of the incentives and whether larger cash payments were associated with stronger program impact. Programs that offered monetary incentives in conjunction with other academic supports tended to show stronger effects than programs that offered incentives alone. Conclusion The findings raise questions as to whether policymakers should use scores from low-stakes test to inform high-stakes policy decisions. The study cautions against using scores from international assessments to rank order countries’ educational systems or using scores from state achievement tests to sanction schools or award teacher bonuses.

The Measure Matters: Examining Achievement Gaps on Cognitively Demanding Reading and Mathematics Assessments

This study presents estimates of Black–White, Hispanic–White, and income achievement gaps using data from two different types of reading and mathematics assessments: constructed‐response assessments that were likely more cognitively demanding and state achievement tests that were likely less cognitively demanding (i.e., composed solely or largely of multiple‐choice items). Specifically, the study utilized multilevel modeling of data from over 25,000 fourth‐ through eighth‐grade students participating in the 6‐state Measures of Effective Teaching (MET) study of 2009–2010, including data from the state reading and mathematics achievement tests used in MET districts at that time and data from the Stanford Achievement Test Open‐Ended Reading Assessment (SAT‐9OE) and the Balanced Assessment of Mathematics (BAM). The latter two assessments, consisting entirely of constructed‐response items, were selected by MET researchers to assess learning outcomes, such as those included in the Common Core State Standards, deemed more cognitively complex than those assessed by state achievement tests at the time. The investigator found that estimated Black–White, Hispanic–White, and income achievement gaps were smaller on the SAT‐9OE than on state reading assessments, before accounting for other relevant factors. Estimates of Black–White and Hispanic–White mathematics achievement gaps were slightly larger using BAM data, whereas the estimated income achievement gap was slightly smaller using BAM data. In later models, prior student academic achievement and average student subject‐specific prior achievement accounted for portions of these estimated achievement gaps.

Relations between CBM (oral reading and maze) and reading comprehension on state achievement tests: A meta-analysis

The purpose of this study was to examine the validity of two widely used Curriculum-Based Measures (CBM) in reading—oral reading and maze—in relation to reading comprehension on state tests using a meta-analysis. A total of 61 studies (132 correlations) were identified across Grades 1 to 10. A random-effects meta-analysis was conducted to estimate the average correlations between the two CBM tasks and reading comprehension on state tests, and to analyze the effects of potential moderating variables (characteristics of study, students, CBM, and state tests). Results revealed that the average correlation for oral reading was significantly larger than that for maze when all grade levels were included together in the analysis. When grade levels were separated, the difference between average correlations was only at the higher grades (Grades 4–10), favoring oral reading. In terms of correlations by grade level, oral reading and maze showed a similar pattern; that is, correlations were comparable across elementary grades, but decreased for secondary grades. In addition to the type of CBM and grade level differences, type of publication, type of state tests (commercial versus state-developed), and time interval between CBM and state tests were significant sources of variance in correlations. Implications for research and educational practice are discussed highlighting the somewhat different conclusions from previous literature, especially regarding the use of CBM for older students.

Effects of New Hampshire’s innovative assessment and accountability system on student achievement outcomes after three years

New Hampshire’s Performance Assessment of Competency Education (PACE) pilot received a waiver from federal statutory requirements related to state annual achievement testing starting in the 2014-15 school year. PACE is considered an “innovative” assessment and accountability system because performance assessments are used to help determine student proficiency in most federally required grades and subjects instead of the state achievement test. One key criterion for success in the early years of the PACE innovative assessment system is “no harm” on the statewide accountability test. This descriptive study examines the effect of PACE on Grades 8 and 11 mathematics and English language arts student achievement during the first three years of implementation (2014-15, 2015-16, and 2016-17 school years) and the extent to which those effects vary for certain student subgroups using results from the state’s accountability tests (Smarter Balanced and SATs). Findings suggest that students in PACE schools tend to exhibit small positive effects on the Grades 8 and 11 state achievement tests in both subjects in comparison to students attending non-PACE comparison schools. Lower achieving students tended to exhibit small positive differential effects, whereas male students tended to exhibit small negative differential effects. Implications for research, policy, and practice are discussed.

Long-Term Validity and Diagnostic Accuracy of a Reading Computer-Adaptive Test

Universal screening is the first mechanism by which students are identified as at risk of failure in the context of multitiered systems of supports. This study examined the validity and diagnostic accuracy of a reading computer-adaptive test as a screener to identify state achievement test performance for third through fifth graders ( N = 1,696). Single time points and slopes within year and longitudinally were examined. Validity results for single points were moderate (.60–.79, p &lt; .002). Validity for slopes and the state test were weak or not significant. Diagnostic accuracy cut scores that maximized sensitivity and specificity yielded high accuracy for single points whereas sensitivity was inadequate for slopes. Practical implications and future directions are presented.