Statewide Achievement Tests Research Articles

A cost-effectiveness analysis of four approaches to universal screening for reading risk in upper elementary and middle school

Research on universal screening in reading has primarily focused on the psychometric adequacy of screening procedures without critically considering costs and value. Educators in upper elementary and middle school have access to a great deal of extant student achievement data, which makes the evaluation of the costs associated with collecting new data for screening purposes paramount. We conducted a retrospective analysis of four approaches to reading screening using cost-effectiveness analysis. Universal screening (i.e., aimswebPlus reading) and statewide reading test data were collected in a midsize school district in Texas for 19,417 students in Grades 4–8. We analyzed the total cost, the classification accuracy based on local cut-scores derived using receiver operating characteristic analysis, and the cost-effectiveness of each approach. Full implementation of the fall administration of aimswebPlus reading cost the district $55,199 in upper elementary and $76,832 in middle school. Both the use of prior-year state achievement test data alone (which represented no additional costs for screening) and the multivariate model (i.e., prior-year state achievement test data plus aimswebPlus) met typical recommendations for classification accuracy. A multiple-gate model that used prior-year state test data and fall aimswebPlus reading on a low-achieving subset of students produced the lowest cost-effectiveness ratio at $156 per additional student accurately classified in upper elementary and $181 in middle school, but this model also led to the highest number of false negatives. The use of aimswebPlus alone resulted in the highest costs and lowest classification accuracy. The results suggest that the use of prior-year statewide achievement test data alone in Grades 4–8 is an efficient approach to universal screening for reading risk that may allow schools to shift resources from screening to other educational priorities.

Predicting performance on a statewide reading achievement test in Spanish with aimswebPlus Spanish

AbstractThe Every Student Succeeds Act requires that states make efforts to offer statewide achievement tests in languages, other than English, that are present to a significant extent in the state. Currently, four states offer Spanish reading or language arts assessments as part of their statewide testing program. In these states, practitioners may be interested in screening students who will complete the Spanish assessments to identify those who are at risk for poor performance. Spanish versions of curriculum‐based measures of reading appear well suited for this purpose. We conducted a retrospective analysis of data collected in a high‐performing district in Texas to investigate the predictive validity and diagnostic accuracy associated with the aimswebPlus Spanish scores when predicting statewide Spanish reading assessment performance for students in Grades 3 through 5. Across grades and seasons, the aimswebPlus Spanish scores explained a small amount of variance in statewide Spanish reading assessment (r 2 = 0.17–0.30). When risk on the aimswebPlus was determined using the national 35th percentile, screening decisions were associated with poor sensitivity (range = 0.50–0.72). Decisions were associated with higher specificity (range = 0.67–0.84) with adequate specificity obtained in four of six comparisons. Limitations, practical implications, and future research directions are discussed.

Longitudinal disciplinary and achievement outcomes associated with school‐wide PBIS implementation level

AbstractPositive behavioral interventions and support (PBIS) is a multitiered framework for behavioral support that is being increasingly implemented in schools. Although research has linked PBIS to improved student outcomes, less research is longitudinal and considers both academic and behavioral outcomes. This study examined the relationships between changes in school‐wide (i.e., Tier I) PBIS implementation level and changes in disciplinary and achievement outcomes over a 2‐year period. Participants included 85 Ohio schools with data on PBIS implementation fidelity (from the tiered fidelity inventory, or TFI), out‐of‐school suspensions (OSSs), and statewide achievement tests for the 2015–2016 and 2016–2017 school years. Linear regressions were conducted to examine the degree to which changes in TFI Tier I scores predicted changes in OSS and achievement scores when controlling for school‐level demographic variables that also were related to the outcome variables. Changes in TFI Tier I scores was found to significantly predict changes in OSS per 100 students but not changes in achievement scores.

Interval likelihood ratios: Applications for gated screening in schools

Educational researchers have traditionally evaluated universal screening performance by dichotomizing continuous test scores to classify students as at-risk or not at-risk. This is pragmatic for decision making, but dichotomization results in a loss of critical information regarding the magnitude of risk. Another approach, which is common in medical research, is to evaluate screening test performance by dividing scores into ordinal categories for which interval likelihood ratios can be estimated. Interval likelihood ratios have yet to be applied to academic screening research. We reanalyzed data from a math screening study in middle school to evaluate differences in screening accuracy and efficiency when dichotomous or interval likelihood ratios were used to interpret screening performance within a gated screening model. Student performance on the prior year statewide achievement test was used as the first gate, followed by six different permutations of student performance on two math curriculum-based measures and the Measures of Academic Progress. Each model was used to predict student proficiency on the subsequent statewide achievement test in math. Treating screening performance as an ordinal variable yielded wider ranges between likelihood ratios than when screening scores were dichotomized. After applying a threshold decision-making model to interpret post-test probability of risk, the minimum number of tests required to classify all students as at-risk or not at-risk ranged between two and three (depending on grade and method of estimating likelihood ratios). The diagnostic accuracy results were similar when interval likelihood ratios or dichotomized results were used to interpret screening scores. Although replication of these findings is needed, one potential benefit of using interval likelihood ratios may be the reduction of the number of students who required additional screening after the first gate.

The Relationship Between School Administrator and Teacher Ratings of Classroom Practices and Student Achievement in High-Poverty Schools

This study examined the relationship of school administrator and teacher self-ratings of instructional and behavioral management practices to student growth on statewide achievement tests (Partnership for Assessment of Readiness for College and Career [PARCC]). The study included 78 teachers and 1,594 students from fourth through eighth grades in nine high-poverty charter schools. Observation scores completed by school administrator and teacher self-ratings were collected on the Classroom Strategies Assessment System (CSAS), an observational assessment that reports outcomes as discrepancy scores: differences between recommended frequency and observed frequency of specific instructional and behavior management strategies for teachers. Correlations revealed negative relations between both informants’ discrepancy scores and PARCC growth scores, demonstrating that teachers with lower discrepancy scores tended to have students with greater PARCC growth scores. Hierarchical multiple regression analyses revealed school administrator and teacher CSAS total discrepancy scores were related to student performance on PARCC mathematics, but not English Language Arts (ELA), and teachers’ CSAS Total discrepancy scores explained an additional 4.8% of variance in PARCC mathematics. Implications of findings for professional development and research are offered.

Co-occurrence of academic and behavioral risk within elementary schools: Implications for universal screening practices.

The purposes of this study were twofold. The first was to use latent class analysis to identify groupings of students defined by the presence or absence of academic or behavioral risk. The second was to determine whether these groups differed across various dichotomous academic and behavioral outcomes (e.g., suspensions, office discipline referrals, statewide achievement test failure). Students (N = 1,488) were sampled from Grades 3-5. All students were screened for academic risk using AIMSweb Reading Curriculum-Based Measure and AIMSweb Mathematics Computation, and behavioral risk using the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS). Latent class analyses supported the fit of a three-class model, with resulting student classes defined as low-risk academic and behavior (Class 1), at-risk academic and high-risk behavior (Class 2), and at-risk math and behavior (Class 3). Logistic regression analyses indicated the classes demonstrated statistically significant differences statewide achievement scores, as well as suspensions. Further analysis indicated that the odds of all considered negative outcomes were higher for both groups characterized by risk (i.e., Classes 2 and 3). Negative outcomes were particularly likely for Class 2, with the odds of negative behavioral and academic outcomes being 6-15 and 112-169 times more likely, respectively. Results were taken to support an integrated approach to universal screening in schools, defined by the evaluation of both academic and behavioral risk. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Investigating the effect of school‐wide positive behavioral interventions and supports on student learning and behavioral problems in elementary and middle schools

AbstractMany states have scaled up School‐Wide Positive Behavioral Interventions and Supports (SW‐PBIS) with the goal of improving student behavior and academic outcomes. Although the effects of SW‐PBIS on behavioral and discipline outcomes have been promising, the findings for academic achievement have been inconclusive and are often limited to cross‐sectional data. This paper examined the longitudinal effect of SW‐PBIS on student behavioral problems and academic achievement growth in elementary and middle schools using propensity score matching. We found no statistically significant longitudinal effects of SW‐PBIS on either student behavior problems or academic achievement among elementary and middle schools in this quasi‐experimental design, even though the positive direction of longitudinal changes would be identified. The result raised questions about the efficacy of SW‐PBIS using a statewide achievement test when scaled up to a state level. Additional research is needed to better understand the impact of SW‐PBIS using rigorous scale‐up designs.

Evaluating Screening Procedures Across Changes to the Statewide Achievement Test

Several states have changed their statewide achievement tests over the past 5 years. These changes may pose difficulties for educators tasked with identifying students in need of additional support. This study evaluated the stability of decision-making accuracy estimates across changes to the statewide achievement test. We analyzed extant data from a large suburban district in Wisconsin in 2014–2015 ( N = 2,774) and 2015–2016 ( N = 2,882). We estimated the decision-making accuracy of recommendations from the Measures of Academic Progress for predicting risk on a Common Core State Standards aligned test (2014–2015) and a new test based on updated academic standards (2015–2016) in reading and math. Findings suggest that sensitivity and specificity estimates were relatively stable in math. Changes in the criterion measure were associated with decreased sensitivity when predicting performance in reading. These results provide initial support for educators to continue existing screening practices until test vendors or state educational agencies establish cut-scores for predicting risk on the newer test. Using a lower cut-score to establish risk (increasing sensitivity while decreasing specificity) may be prudent in reading. Limitations and directions for future research are discussed.

Technical Adequacy of the Social, Academic, and Emotional Behavior Risk Screener in an Elementary Sample

The purpose of this study was to evaluate the psychometric defensibility of the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS): a quick and easy universal screener for behavioral and emotional risk. Elementary school teachers completed the SAEBRS with 346 students in Grades 3 to 5. Teachers also completed two criterion measures, including the Student Risk Screening Scale (SRSS) and the Student Internalizing Behavior Screener (SIBS). Additional extant behavioral and academic data sources were collected including office discipline referrals, suspensions, curriculum-based measurement scores, and statewide achievement test scores. Reliability analyses were indicative of the internal consistency of all four SAEBRS scales, whereas correlational analyses and Mann–Whitney–Wilcoxon tests supported the criterion-related and construct validity. Receiver operating characteristic curve analyses suggested each SAEBRS scale was associated with acceptable or optimal diagnostic accuracy. However, cut scores selected as most appropriate within each SAEBRS scale were found to differ from those identified in previous studies, potentially suggesting the influence of criterion outcome under consideration on SAEBRS diagnostic accuracy. Limitations and future directions for research are discussed, with emphasis on the need for continued examination of the extent of variability in SAEBRS cut score performance.

Using latent class analysis to identify academic and behavioral risk status in elementary students.

Identifying classes of children on the basis of academic and behavior risk may have important implications for the allocation of intervention resources within Response to Intervention (RTI) and Multi-Tiered System of Support (MTSS) models. Latent class analysis (LCA) was conducted with a sample of 517 third grade students. Fall screening scores in the areas of reading, mathematics, and behavior were used as indicators of success on an end of year statewide achievement test. Results identified 3 subclasses of children, including a class with minimal academic and behavioral concerns (Tier 1; 32% of the sample), a class at-risk for academic problems and somewhat at-risk for behavior problems (Tier 2; 37% of the sample), and a class with significant academic and behavior problems (Tier 3; 31%). Each class was predictive of end of year performance on the statewide achievement test, with the Tier 1 class performing significantly higher on the test than the Tier 2 class, which in turn scored significantly higher than the Tier 3 class. The results of this study indicated that distinct classes of children can be determined through brief screening measures and are predictive of later academic success. Further implications are discussed for prevention and intervention for students at risk for academic failure and behavior problems. (PsycINFO Database Record

Predictive Utility and Classification Accuracy of Oral Reading Fluency and the Measures of Academic Progress for the Wisconsin Knowledge and Concepts Exam

This study examined the predictive validity and classification accuracy of two commonly used universal screening measures relative to a statewide achievement test. Results indicated that second-grade performance on oral reading fluency and the Measures of Academic Progress (MAP), together with special education status, explained 68% of the variance in third-grade standardized test performance ( N = 399). Six methods of dichotomous risk classification were also explored to predict whether students would pass or fail the state test. A random sample of half of the participants was used to establish locally appropriate risk cut-scores, holding sensitivity constant at .95; the cut-scores were then cross-validated on the remaining participants. Cross-validation reflected sensitivity of above .90 for all methods, with MAP Language Usage and two multi-screener methods demonstrating the strongest overall technical adequacy for risk classification. The limitations and implications of these findings are discussed.

Growth and gaps in mathematics achievement of students with and without disabilities on a statewide achievement test

This study estimated mathematics achievement growth trajectories in a statewide sample of 92,045 students with and without disabilities over Grades 3 to 7. Students with disabilities (SWDs) were identified in seven exceptionality categories. Students without disabilities (SWoDs) were categorized as General Education (GE) or Academically/Intellectually Gifted (AIG). Students in all groups showed significant growth that decelerated over grades as well as significant variability in achievement by student group, both at the initial assessment in Grade 3 and in rates of growth over time. Race/ethnicity, gender, parental education, free/reduced lunch status, and English language proficiency were also significant predictors of achievement. Effect size estimates showed substantial year-to-year growth that decreased over grades. Sizeable achievement gaps that were relatively stable over grades were observed between SWoDs and students in specific exceptionality categories. Our study also demonstrated the importance of statistically controlling for variation related to student demographic characteristics. Additional research is needed that expands on these results with the same and additional exceptionality groups.

Comparing students’ self-discipline and self-regulation measures and their prediction of academic achievement

Using a multi-source, multi-measure research design involving 507 high school students and their teachers, we compared prediction of these students’ academic achievement by a composite of students’ and teachers’ measures of students’ self-regulation (SR) with a composite of students’ and teachers’ measures of students’ self-discipline (SD). Hierarchical regression analyses revealed that the SR composite was more predictive of students’ grade point average and performance on a state-wide achievement test than the SD composite. Confirmatory factor analyses showed that, although SD and SR latent factors correlated significantly, a two-factor solution provided an acceptable fit for the results. Structural Equation Modeling analyses indicated that the SR latent factor predicted both measures of students’ achievement significantly, but the SD factor did not predict either achievement measure significantly. No significant gender differences were found with students’ SD, SR, or achievement measures. These results suggest a path for integrating two relatively separate streams in self-regulation research on the basis of a well-established distinction between learning and performance processes.

Relationships Among Item Cognitive Complexity, Contextual Demands, and Item Difficulty: Implications for Achievement-Level Descriptors

A primary goal of standards-based statewide achievement tests is to classify students into achievement levels that enable valid inferences about student content area knowledge and skill. Explicating how knowledge and skills are expected to differ in complexity in achievement level descriptors, and how that complexity is related to empirical item difficulty—at the beginning of assessment design, rather than at the end—is critical to those inferences. The purpose of this study is to investigate whether sources of knowledge and skill descriptions that are commonly found in achievement-level descriptors relate to item complexity and difficulty. Implications of findings for using achievement-level descriptors as a method of interpreting test scores and for test design are discussed.

Predicting End-of-Year Achievement Test Performance

Motivated by the multiple-measures clause of recent federal policy regarding student eligibility for alternate assessments based on modified academic achievement standards (AA-MASs), this study examined how scores or combinations of scores from a diverse set of assessments predicted students’ end-of-year proficiency status on statewide achievement tests. Scores from 388 eighth-grade students, both with and without disabilities, from 3 states were compared for the predictive validity of test inferences from a set of brief online screening tests, teachers’ ratings of performance, and students’ previous year’s achievement tests. The research team developed screening tests in reading and mathematics that were found to be internally consistent and added predictive value to the identification of students who would perform at a level below proficient on state achievement tests of reading and mathematics. Although achievement results from the previous year were more predictive of future achievement results, scores from the online screening tests and from Performance Screening Guides could be used as supplements to typical achievement tests. Implications of these findings for educators in the 17 states currently using AA-MAS tests for students with disabilities are discussed, and needed future research is identified.

Coming to Know: One Teacher Widens and Deepens Her Knowledge about Struggling Students

Background As part of their graduate studies, university students seeking a master's degree in language and education enroll in a graduate reading practicum held during the university summer session. In an arrangement with a local public elementary school, practicum participants worked on the school's campus in a classroom setting and with struggling readers. Children were so designated by their classroom performance but more so by their failure to reach a passing score on the statewide achievement test in reading. As a way to offer additional help in reading and boost their achievement level, struggling readers were required to attend summer school. Graduate practicum participants provided some of the necessary instructional help that was offered. This arrangement became the means not only to aid struggling readers but also to enable the graduate students to gain further insights into the teaching of reading. Graduate students read the selected course materials--all related to effective theory and practice as well as classroom assessment theory and measures. The readings and the work of the session called attention to thoughtful literacy (Allington, 2009, p. 116). Informed by the readings, the graduate students had the opportunity to teach reading approximately 90 minutes a day for an entire five-week summer session. Following the daily teaching, the university professors in charge of the practicum and the graduate students gathered together for debriefing sessions. The conversations centered on the reading instruction that was implemented and the children's responses. These daily sessions were a venue for collective reflections and an opportunity for graduate peers to aid each other in further planning of the on-going reading instruction and increasing their knowledge about the teaching of reading. The graduate students also wrote individual reflective journals based on their work with the children and relative to the assigned readings. The daily journal entries further served as an important springboard for discussions during the debriefing sessions. In sum, knowledge gained from the readings, which served to inform the instruction offered to children in combination with the insights of the collective debriefings and the reflective writing, would enable the graduate practicum students to come to know more about teaching reading. Specifically, they would come to know more about teaching struggling readers. Framework for the Study In light of the context described at the outset of this article, one of the goals of the practicum experience was to think about countering what is often the instructional diet provided to struggling readers. As Allington (2009) describes the situation, teachers create instructional environments, even interventions, whereby struggling readers work on reading skills and not on actual reading. According to Allington (2006), teachers need to think about thoughtful literacy (p. 116), whereby students read and discuss with a focus on comprehension that results in higher student achievement. The summer practicum challenged the graduate students to provide the children with reading experiences that would be characterized as thoughtful literacy. To aid in accomplishing the latter, a reading workshop format, with its focus on ample opportunity for actual reading, undergirded the daily instruction for children (Roller, 1996; Serafini, 2001, 2006; Tompkins, 2010). Books with a range of reading levels were made available to children, children self-selected books to read, and personal responses to the readings--written and oral--were encouraged and often shared with peers and graduate students alike. Teacher/student conferences to discuss children's readings were a daily occurrence. Allington's thoughtful literacy thus played out in lesson preparation and in the choice of instructional materials. The complexities of teacher education and published calls for reform in teacher education are well documented (e. …

A Preview of the Special Issue

Within professional circles, conversations about educational services and supports for students with disabilities increasingly reference the extent to which compelling research support exists for particular educational practices. Although individual studies certainly provide important insights into the delivery and impact of educational practices, it is the accumulation of research findings across multiple studies through which evidence-based strategies are established. Systematic literature reviews play a vital role in informing practitioners about whether and how a particular practice affects particular outcomes for students with disabilities. Moreover, such analyses can be instrumental in identifying critical gaps in the research literature and the need for methodological innovations. This special issue represents a collection of seven systematic literature reviews addressing a range of important issues in the education of students with disabilities and students at risk for school failure. In the first review, Yeo reports findings from a meta-analysis of 27 studies addressing curriculum-based measurement and statewide achievement tests. Findings suggest that curriculum-based measures may be valid for predicting performance on statewide reading tests. However, this association appears to be moderated by several factors. In the second review, Berkeley, Scruggs, and Mastropieri report on their analysis of 40 studies focused on improving the reading comprehension of students with learning disabilities. Collectively, findings from these studies reaffirm the effectiveness of an array of intervention strategies for improving student performance on both criterion- and normreferenced measures. The third and fourth reviews focus on the efficacy of interventions for secondary-age students with disabilities. Scruggs, Mastropieri, Berkeley, and Graetz synthesize findings from 70 intervention studies focused on improving outcomes for middle and high school students with disabilities within the content areas of English, science, and social studies. In this meta-analysis, the authors identify a number of intervention approaches—including explicit instruction, mnemonic instruction, peer instruction, and computer-assisted instruction—associated with moderate to large effect sizes. Okilwa and Shelby examine the effectiveness of peer tutoring interventions on the academic outcomes of youth with disabilities. Findings from these 12 studies highlight the efficacy of this educational practice across a range of students, settings, and academic measures. In the fifth review, the focus shifts to the other end of the educational spectrum. Namely, Maag and Katsiyannis focus their review on an appraisal of the research support available for four early intervention programs designed for children with or at risk for antisocial behavior. In their analysis, they address the different levels of evidence for the efficacy of each program and raise several critical issues associated with the evaluation of early intervention efforts. In the sixth review, Hart and Banda review 13 single-case intervention studies examining the impact of the Picture Exchange Communication System (PECS) on the functional communication of students with intellectual and developmental disabilities. Although the PECS approach was considered moderately to highly effective for some communication outcomes, the authors identify several key areas in which additional research is needed to strengthen the evidence base. In the final article, Lindo and Elleman review the literature on reading interventions to ascertain the extent to which the views of teachers and students were solicited in these studies. The authors find that the perspectives of these stakeholders were infrequently sought. This is an interesting review, given that concerns about the research to practice gap often revolve around issues of social validity These seven literature reviews reflect a range of approaches for identifying, coding, interpreting, and evaluating a particular segment of the research literature. We hope this special issue will provide valuable insights into the impact of a variety of educational interventions. Moreover, we anticipate the articles will prompt further discussion of how best to synthesize the rapidly expanding special education research base. 386556

A Comparison of Approaches for Improving the Reliability of Objective Level Scores

This study is an evaluation of the psychometric issues associated with estimating objective level scores, often referred to as ‘‘subscores.’’ The article begins by introducing the concepts of reliability and validity for subscores from statewide achievement tests. These issues are discussed with reference to popular scaling techniques, classical test theory, and item response theory. Methods for increasing the reliability of subscore estimates that have been suggested in literature are then reviewed. Based on this review, an empirical study comparing some of the more promising procedures was conducted. Test score data from a large statewide testing program were analyzed in this study. The comparison of subscore augmentation approaches found that generally all methods were very successful in dramatically increasing the reliability of subscore estimates. However, this increase was accompanied by near-perfect correlations among the subscore estimates. This finding called into question the validity of the resultant subscores, and therefore the usefulness of the subscore augmentation process. Implications for practice are discussed.

Predicting Performance on State Achievement Tests Using Curriculum-Based Measurement in Reading: A Multilevel Meta-Analysis

The purpose of this synthesis was to examine the relationship between Curriculum-Based Measurement (CBM) and statewide achievement tests in reading. A multilevel meta-analysis was used to calculate the correlation coefficient of the population for 27 studies that met the inclusion criteria. Results showed an overall large correlation coefficient of .689, which was heterogeneous among studies. Study sample size, proportion of English language learners, proportion of students with disabilities, and difference in time from when CBM and statewide achievement tests were administered were statistically significant components affecting the relationship between CBM and state achievement tests. Limitations, educational implications, and directions for future research are presented.

Long-Term Diagnostic Accuracy of Curriculum-Based Measures in Reading and Mathematics

.This study examined the relation between benchmark data and rate of growth across the year for reading, math computation, and math concepts and applications curriculum-based measures, and a statewide and large-scale achievement test. Results showed that reading and math benchmarks were significantly and moderately correlated with performance on both achievement measures 1 and 2 years later. Rate of growth in first-grade reading was significantly and moderately related to third-grade statewide achievement performance, with the relation decreasing to nonsignificance by Grade 3. Math computation growth was also related to performance on the statewide achievement test. Curriculum-based measures provided adequate diagnostic accuracy for screening of performance on statewide and large-scale achievement tests both 1 and 2 years later. These data suggest the importance of early identification of students who are at risk for developing academic problems in reading and math.