Universal Screener Research Articles

Reliability and validity of a Chinese language universal behavioral screener

ABSTRACT This study examines the reliability and validity of the Chinese language version of the Integrated Screening and Intervention System Teacher Rating Form (ITRF) as a screening tool for problem behaviors among secondary school students in Hong Kong. The Chinese ITRF was validated through identifying factor structure using exploratory factor analysis and evaluating internal consistency, temporal stability, convergent validity, and discriminant validity. Thirty-one teachers completed the ITRF and Strengths and Difficulties Questionnaire (SDQ) for five students nominated for behavioral problems and five randomly chosen students in their class with a subset of 27 teachers repeating the procedure two to five weeks later for test–retest analysis. Results support a two-factor structure (i.e., Oppositional/Disruptive, Academic Productivity/Disorganization) as identified in the original ITRF and German ITRF. The Chinese ITRF demonstrated adequate internal consistency (αs > .90) and temporal stability (rs > .80). Correlation coefficients between similar (rs = .54–.74) and dissimilar (rs = .06–.24) ITRF and SDQ subscales supported its convergent validity and discriminant validity, respectively. The findings imply the measurement invariance of the ITRF and usability of the Chinese ITRF as a validated universal screener for behavior.

Integrating Validation Arguments With the Assessment Triangle: A Framework for Operationalizing and Instantiating Validation

ABSTRACTDespite the call for an argument-based approach to validity over 25 years ago, few examples exist in the published literature. One possible explanation for this outcome is that the complexity of the argument-based approach makes implementation difficult. To counter this claim, we propose that the Assessment Triangle can serve as the overarching framework for operationalizing and instantiating the argument-based approach to validation. Integrating these frameworks can streamline the validation process by providing a conceptual lens for identifying, collecting, and evaluating relevant sources of evidence throughout the testing process. To fully examine this proposed conceptualization, we apply the integrated framework to an example case of a universal screener for middle-school mathematics. We articulate an interpretation and use argument for the universal screener, and then present relevant sources of evidence to evaluate the plausibility of the inferences and warrants underlying test score use. Based on this applied example, the strengths and limitations of the integrated framework are considered, and recommendations are made for future instantiations.

Initial Validation of an Elementary Version of the Positive Family Support Strengths and Needs Assessment

The purpose of the present study was to conduct an initial validation of the Positive Family Support–Strengths and Needs Assessment (PFS-SaNA) for elementary school children. The PFS-SaNA is designed as a universal screener wherein parents report on common areas their children may need additional support. Parents of children attending one elementary school participated in two consecutive years. A total of 627 parents of elementary school children participated (Year 1 = 368, Year 2 = 259). Exploratory factor analysis was conducted across Years 1 and 2 and a replicability test compared item loadings from Year 1 and Year 2. Finally, evidence of external validity was examined. Results suggested the PFS-SaNA is unidimensional with findings indicating item loadings replicated from Year 1 to Year 2. Finally, evidence of external validity was found. Implications for the PFS-SaNA as a universal parent screener that can be embedded in a multiple gating risk management strategy in elementary school are discussed.

Moving the Needle on Evaluating Multivariate Screening Accuracy

The assessment of screening accuracy and setting of cut points for a universal screener have traditionally been evaluated using logistic regression analysis. This analytic technique has been frequently used to evaluate the trade-offs in correct classification with misidentification of individuals who are at risk of performing poorly on a later outcome. Although useful statistically, coefficients from a multiple logistic regression can be difficult to explain to practitioners as it pertains to classification decisions. Moreover, classifications based on multivariate assessments are challenging to understand how performance on one assessment compensates for performance on another. The purpose of this article is to demonstrate and compare the use of logistic regression with classification and regression tree (CART) models in the identification of students who are at risk of reading comprehension difficulties. Data consisted of 986 Grade 1 students and 887 Grade 2 students who were administered a screening assessment at the middle of the school year as well as the 10th edition of the Stanford Achievement Test. Results indicated that CART performs comparably with logistic regression and may assist researchers and practitioners in explaining classification rules to parents and educators.

Measuring Teachers’ Perceptions of Student Behavior Using the Systematic Screening for Behavior Disorders in Middle School Students

Universal screening serves to identify students at risk for emotional and behavioral difficulties while their behavior is still amenable to treatment. However, there are few universal screeners available for middle school students who may be at heightened risk for developing emotional and behavioral difficulties due to major academic and social challenges. The Systematic Screening for Behavior Disorders (SSBD), often considered the gold standard for behavior screeners, was recently validated for use in middle schools. However, there is little research on the reliability and validity of the SSBD for screening middle school students and the differences among teachers of different academic subjects. The purpose of this study was to extend the previous research validating the SSBD assessing its technical adequacy and sensitivity in middle school students among teachers of different academic subjects. Results indicated that the SSBD provided reliable and valid evidence for identifying at-risk middle school students. This study provided evidence that the SSBD identified internalizing and externalizing students with a similar degree of accuracy.

Measurement Invariance of a Universal Behavioral Screener Across Samples From the USA and Germany

Abstract. The current study examines the item and scalar equivalence of an abbreviated school-based universal screener that was cross-culturally translated and adapted from English into German. The instrument was designed to assess student behavior problems that impact classroom learning. Participants were 1,346 K-6 grade students from the US (n = 390, Mage = 9.23, 38.5% female) and Germany (n = 956, Mage = 8.04, 40.1% female). Measurement invariance was tested by multigroup confirmatory factor analysis (CFA) across students from the US and Germany. Results support full scalar invariance between students from the US and Germany (df = 266, χ2 = 790.141, Δχ2 = 6.9, p < .001, CFI = 0.976, ΔCFI = 0.000, RMSEA = 0.052, ΔRMSEA = −0.003) indicating that the factor structure, the factor loadings, and the item thresholds are comparable across samples. This finding implies that a full cross-cultural comparison including latent factor means and structural coefficients between the US and the German version of the abbreviated screener is possible. Therefore, the tool can be used in German schools as well as for cross-cultural research purposes between the US and Germany.

Universal screening for algebra readiness in middle school: Why, what, and does it work?

ABSTRACTWithin a multi-tier system of support (MTSS), results from universal screeners help teachers make instructional decisions early in the learning process to prevent and remediate skill gaps. Universal screeners help teachers determine students’ need for and intensity of additional instructional support to reach their learning goals by efficiently providing reliable and trustworthy information. In this article, we describe the value of implementing universal screening assessments in middle school mathematics that focus on algebra readiness. To instantiate a framework for developing and evaluating an algebra-readiness universal screener, we present an example universal screener and discuss the design considerations used to create the assessment system and share a summary of technical adequacy data to document the utility of the results for supporting teachers’ decision making. This example is intended to illustrate the framework to support test developers and administrators who are interested in providing teachers with useful data about students’ readiness for algebra.

Monitoring and Fostering Elementary School Students' Life Satisfaction: A Case Study

ABSTRACTResearch on the importance of complete mental health has led to increased focus on students' subjective well-being (i.e., happiness and life satisfaction) coupled with the traditional attention to psychopathology. Although screeners for psychopathology abound, the present project is novel in applying best practices in universal screening to grade level-wide assessment of children's life satisfaction, followed by implementation of a positive psychology intervention for students identified through the screener. In this case study of 375 third- to fifth-grade students in one elementary school, 93.6% participated in the screening, 11.2% qualified for supplemental services based on screening cutoff scores, and 4.8% took part in the targeted intervention to improve subjective well-being. Repeated measures analysis of mean scores revealed the intervention increased children's life satisfaction, lending support for the value of universal screeners in the process of service delivery. Implications for practice regarding universal screening and intervention for life satisfaction are discussed.

On Using Simulations to Inform Decision Making During Instrument Development

Although content validity—how well each item of an instrument represents the construct being measured—is foundational in the development of an instrument, statistical validity is also important to the decisions that are made based on the instrument. The primary purpose of this study is to demonstrate how simulation studies can be used to assist the decision making of researchers who are developing or updating instruments. A hypothetical research study is presented in which the researcher needs to make choices that will guide his or her development of a universal behavior screener. In the study, he or she wishes to make choices regarding the number of items to include, the pilot sample size, average difficulty of the items, and the amount of information provided by the instrument at different cut scores. Simulation is then used to create data sets with varying levels of each of these aspects, and decisions are subsequently made regarding the levels that should be applied in the actual study. Rationale for these decisions as well as implications for practice are included.

The Student Risk Screening Scale: A Reliability and Validity Generalization Meta-Analysis

The purpose of this study was to conduct reliability and validity generalization meta-analyses of evidence regarding the Student Risk Screening Scale (SRSS), a universal screener for externalizing behavior problems. A systematic review of the literature resulted in the identification of 17 studies inclusive of evidence regarding SRSS score (a) internal consistency reliability (i.e., alpha coefficients), and/or (b) criterion-related validity (e.g., correlations between the SRSS and various outcomes). Multilevel meta-analyses indicated that across studies, SRSS scores were associated with adequate internal consistency (α = .83). Analyses further suggested the SRSS was a valid indicator of both social and behavioral outcomes ( r = .52) and academic outcomes ( r = .42). Follow-up analyses suggested that in accordance with theory-driven expectations, the SRSS was a stronger indicator of externalizing problems and broad behavior outcomes relative to alternative outcomes (e.g., internalizing problems). Limitations and directions for future research are discussed, including recommendations for the collection of additional SRSS diagnostic accuracy evidence.

Screening for Student Mental Health Risk: Diagnostic Accuracy, Measurement Invariance, and Predictive Validity of the Social, Academic, and Emotional Behavior Risk Screener-Student Rating Scale (SAEBRS-SRS)

Schools have become the largest provider of mental health services to children, thus increasing the need to identify children at risk before problems worsen. The Social, Academic, and Emotional Behavioral Risk Screener-Student Rating Scale (SAEBRS-SRS) student version is a universal screener that assesses pre-symptomology of internalizing and externalizing behaviors. Prior research on the SAEBRS-SRS has supported a psychometrically defensible factor structure; however, additional research is necessary to support applied used in schools. In this study, 1102 middle grades students completed the SAEBRS-SRS and the Strengths and Difficulties Questionnaire. Data were analyzed to evaluate the diagnostic accuracy of the screener via receiver operating characteristic curve analyses. Multi-group confirmatory factor analysis was conducted to identify measurement invariance across gender. Results indicated that (1) the SAEBRS-SRS has adequate diagnostic accuracy statistics, particularly for the SAEBRS Total score, and (2) the SAEBRS-SRS is invariant across gender at the metric and scalar levels of measurement invariance. Implications and recommendations for future research are further discussed.

The Behavioral and Emotional Screening System Student Form: Is There Evidence of a Global At-Risk Factor in a Sample of Predominantly African American Youth?

School-based universal screening for behavioral and emotional risk can serve as a foundation for effective multi-tiered service delivery systems. The current study examines the measurement and structure of one such universal screener, the Behavioral and Emotional Screening System Student Form (BESS SF). Four models were investigated including a unidimensional model, a multidimensional model, a second-order model, and a bifactor model. This study is the first to use a bifactor model in examining the structure of the BESS SF, and further adds to the literature by using bifactor modeling with a predominantly low-income, urban, African American sample. Results indicate that the bifactor model provided the best fit. Practically, results support interpretation of the global risk score as well as the domain-specific factors.

Development and Validation of the Social, Academic, and Emotional Behavior Risk Screener–Student Rating Scale

School systems are the primary providers for the increasing number of children with mental health needs. School-based universal screening offers a valuable way to identify children that would benefit from school-based mental health services. However, many existing screening systems rely on teacher ratings alone and do not incorporate student self-ratings. The current study evaluates the psychometric properties of the Social, Academic, and Emotional Behavior Risk Screener–Student Rating Scale (SAEBRS-SRS), a new 20-item multidimensional universal screener intended to provide assessment data on students’ social, academic, and emotional functioning. The SAEBRS-SRS complements the SAEBRS Teacher Rating Scale (TRS), which has previously demonstrated robust psychometric evidence. In the current study, data were collected from a racially and ethnically diverse sample of middle school students. Confirmatory factor analyses supported a bifactor structure consistent with the SAEBRS-TRS, with items corresponding to internally consistent Social, Academic, and Emotional Behaviors subscales, as well as an overall Total Behavior scale. The current analyses yield promising initial support for the development of the SAEBRS-SRS. Implications and the need for future research to provide additional psychometric evidence are discussed.

Latent Structure of Scores From the Emotional and Behavioral Screener

The Emotional and Behavioral Screener (EBS) is a recently developed teacher-reported brief screening instrument for identifying students who are at-risk of an emotional or behavioral disorder (EBD). Although prior research supports the technical adequacy of scores from the EBS, there is a gap in the literature regarding strong evidence of the factor structure underlying EBS scores. This study investigated the latent structure of scores from the EBS in a sample of 646 elementary students who were rated by their teachers in a 2-week screening period. Single-factor confirmatory factor analysis (CFA) and bifactor models were used to test the hypothesis that EBS scores are a measure of both overall emotional and behavioral risk and students’ externalizing and internalizing behaviors. Results supported a bifactor structure, in that scores from the EBS can be considered to represent a general factor (i.e., risk of EBD) and two group factors (i.e., externalizing and internalizing domains). Findings have implications for interpreting scores when using the EBS as a universal screener for the risk of EBD.

Evaluating the Interpretations and Use of Curriculum-Based Measurement in Reading and Word Lists for Universal Screening in First and Second Grade

.Universal screening in elementary schools often includes administering curriculum-based measurement in reading (CBM-R); but in first grade, nonsense word fluency (NWF) and, to a lesser extent, word identification fluency (WIF) are used because of concerns that CBM-R is too difficult for emerging readers. This study used Kane's argument-based approach to validation as a framework to evaluate the interpretations and use of scores resulting from screening 257 first- and second-grade students. First, scores from three word lists (decodable WIF, high-frequency WIF, and whole-word NWF) were examined as indicators of reading achievement. Then, the use of these word list scores was evaluated regarding their ability to classify at-risk readers accurately and as supplements to CBM-R during the winter universal screening period. Participants were also concurrently administered a norm-referenced measure of early reading skills and global reading achievement. Results suggested that the word lists were good indicators of reading achievement and provided support for using CBM-R or a word list in conjunction with CBM-R to discriminate among at-risk readers. Findings have implications for the administration of universal screeners in first and second grade.

Technical Adequacy of the Social, Academic, and Emotional Behavior Risk Screener in an Elementary Sample

The purpose of this study was to evaluate the psychometric defensibility of the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS): a quick and easy universal screener for behavioral and emotional risk. Elementary school teachers completed the SAEBRS with 346 students in Grades 3 to 5. Teachers also completed two criterion measures, including the Student Risk Screening Scale (SRSS) and the Student Internalizing Behavior Screener (SIBS). Additional extant behavioral and academic data sources were collected including office discipline referrals, suspensions, curriculum-based measurement scores, and statewide achievement test scores. Reliability analyses were indicative of the internal consistency of all four SAEBRS scales, whereas correlational analyses and Mann–Whitney–Wilcoxon tests supported the criterion-related and construct validity. Receiver operating characteristic curve analyses suggested each SAEBRS scale was associated with acceptable or optimal diagnostic accuracy. However, cut scores selected as most appropriate within each SAEBRS scale were found to differ from those identified in previous studies, potentially suggesting the influence of criterion outcome under consideration on SAEBRS diagnostic accuracy. Limitations and future directions for research are discussed, with emphasis on the need for continued examination of the extent of variability in SAEBRS cut score performance.

The Kindergarten Academic and Behavior Readiness Screener: The utility of single-item teacher ratings of kindergarten readiness.

The purpose of the study was to explore the effectiveness of a brief, feasible, and cost-effective universal screener for kindergarten readiness. The study examined whether teacher ratings of kindergarteners' academic, behavioral, and overall readiness at the beginning of the year were predictive of academic, emotional, and behavioral outcomes at the end of the year. Participants included 19 kindergarten teachers and their students (n = 350) from 6 urban elementary schools; all teachers were female and the majority of children were African American (74%) or White (23%). Thirty-six percent of children qualified for free or reduced lunch. Teachers completed single-item ratings of student readiness as well as full scale ratings of student prosocial skills, disruptive behaviors, and academic competence. Students also completed a standardized academic achievement test. Independent observers rated disruptive behaviors in the classroom. Readiness items had statistically significant relations with a range of academic, emotional, and behavior indicators. Hierarchical linear regression analyses found that readiness items predicted end-of-year outcomes when controlling for baseline covariates. Items also predicted higher likelihood of negative academic and behavior categorical outcomes and demonstrated classification utility. Schools need universal screening options that are feasible and easy to implement school-wide. The screening tool presented in this study offers a viable, psychometrically strong option for school teams and professionals interested in universal screening.

Peer-mediated check-in/check-out for students at-risk for internalizing disorders.

The present study investigated the effectiveness of peer-mediated check-in/check-out (CICO) on the internalizing behaviors of elementary school students. A nonconcurrent multiple-baseline design across participants was utilized to evaluate the intervention's effectiveness for 3 students in 1st and 2nd grade. Two 5th grade students were trained to implement CICO under the supervision of an adult intervention specialist. The peer-mediated CICO procedure was effective for 2 of the 3 participants as evidenced by moderate to large effect sizes; however, all 3 participants were identified as "at-risk" on a universal screener for internalizing problems. The results suggest peer-mediated CICO may be a resource-efficient Tier II strategy to meet the needs of students engaging in internalizing behavior within a multitiered framework of service delivery.

Measuring annual growth using written expression curriculum-based measurement: An examination of seasonal and gender differences.

The purpose of this study was to examine annual growth patterns and gender differences in written expression curriculum-based measurement (WE-CBM) when used in the context of universal screening. Students in second through fifth grade (n = 672) from 2 elementary schools that used WE-CBM as a universal screener participated in the study. Student writing samples were scored for production-dependent, production-independent, and accurate-production indicators. Results of latent growth models indicate that for most WE-CBM outcome indicators across most grade levels, average growth was curvilinear, with increasing curvilinearity on all indicators as grade level increased. Evidence of gender differences was mixed with girls having higher initial scores on all WE-CBM indicators except for total words written (second and third grades), correct minus incorrect writing sequences (fourth grade only), and percent correct writing sequences (second-fourth grades) where differences were not statistically significant. Despite differences in initial level, there were few gender differences in growth and limited overall between-student variability in linear slope. The results of this study extend research on annual patterns of growth and gender differences in WE-CBM by analyzing all 3 types of WE-CBM indicators, including upper elementary grades, and assessing skills more frequently (i.e., 4 to 5 times in 1 year) than in prior research on annual growth. The findings have implications for universal screening in WE-CBM and for understanding gender differences in writing performance.

Demographic Characteristics Associated with Homelessness and Risk Among Female and Male Veterans Accessing VHA Outpatient Care

BackgroundThis study explored demographic influences on veterans' reports of homelessness or imminent risk of homelessness with a particular focus on gender. MethodsWe analyzed data for a cohort of veterans who responded to the U.S. Department of Veterans Affairs (VA), Veterans Health Administration (VHA) universal screener for homelessness and risk during a 3-month period. Multinomial mixed effects models—stratified by gender—predicted veterans' reports of homelessness or risk based on age, race, marital status, and receipt of VA compensation. FindingsThe proportion of positive screens—homelessness or risk—was 2.7% for females and 1.7% for males. Women more likely to report being at risk of homelessness were aged 35 to 54 years, Black, and unmarried; those more likely to experience homelessness were Black and unmarried. Among male veterans, the greatest predictors of both homelessness and risk were Black race and unmarried status. Among both genders, receiving VA disability compensation was associated with lesser odds of being homeless or at risk. ConclusionsThe findings describe the current population of veterans using VHA health care services who may benefit from homelessness prevention or intervention services, identify racial differences in housing stability, and distinguish subpopulations who may be in particular need of intervention. Interventions to address these needs are described.