Direct Behavior Rating Single Item Scales Research Articles

Exploring threats to internal validity of direct assessment in single-case design research.

Single-case design research studies have historically used external observers to collect time series data that may be used to evaluate intervention effectiveness; however, single-case interventions implemented in educational settings may use the person implementing the intervention (e.g., teacher) to collect data in order to maximize feasibility. The implementer's knowledge of intervention goals and phase has the potential to influence assessment of dependent variables, particularly when ratings involving some degree of judgment (e.g., Direct Behavior Rating-Single Item Scales [DBR-SIS]) are used. Given the potential for rater effects and expectancy to influence data collection, this study sought to determine whether DBR-SIS measuring social skills collected in vivo by interventionists with full knowledge of intervention goals and phase were equivalent to data collected by external raters masked to intervention phase. Results indicated in vivo DBR-SIS differed from those completed by masked external raters, which has the potential to result in different conclusions regarding intervention effectiveness. The potential for negative effects resulting from sole reliance on in vivo ratings conducted by an interventionist may be mitigated by including additional data streams collected by external personnel masked to intervention phase or by using effect sizes that account for baseline trends. Implications for training and practice are discussed. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Timing and frequency of screening in schools: A latent variable analysis of behavioral stability over time.

The growth of school-based initiatives incorporating multitiered systems of support (MTSS) for social, emotional, and behavioral domains has fueled interest in behavioral assessment. These assessments are foundational to determining risk for behavioral difficulties, yet research to date has been limited with regard to when and how often to administer them. The present study evaluated these questions within the framework of behavioral stability and examined the extent to which behavior is stable when measured by two school-based behavioral assessments: the Direct Behavior Rating-Single-Item Scales (DBR-SIS), and the Behavioral and Emotional Screening System (BESS). Participants included 451 students rated three times per year across 4 years, with the primary teacher from each year providing the within-year ratings. Latent variable models were employed to measure the constructs underlying the observed assessment scores. Models demonstrated that the DBR-SIS best captured changes within the year, whereas the BESS scores remained stable across time points within a year. Across years, scores from both assessments captured changes. The unique contributions of each assessment in the data-based decision-making process are discussed, and recommendations are given for their combined use within and across school years. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

The effects of an online DBR training module on rater accuracy.

A multiple baseline design across four teachers was used to investigate the effects of an online Direct Behavior Rating (DBR) training module on rater accuracy of academic engagement, respect, and disruptive behavior. Rater accuracy was examined by comparing Direct Behavior Rating-Single Item Scales (DBR-SIS) completed by classroom teachers to systematic direct observation. Teacher acceptability of DBR-SIS was also investigated. Teachers rated the use of DBR-SIS as a formative evaluation tool favorably. Rater accuracy of DBR-SIS was generally higher after teachers completed the online training module. However, a visual analysis of the data revealed no clear impact of the online training. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Dependability of Direct Behavior Rating Single- and Multi-Item Scales Across Raters and Occasions in Two School Subjects

Direct Behavior Rating (DBR) has been proposed as a promising approach to assess student behavioral progress in classroom settings. The current study examines raters and occasions as well as their interactions as sources of error in the use of DBR–Multi-Item Scales (DBR-MIS) and DBR–Single-Item Scales (DBR-SIS) for academically engaged (AE) and disruptive behavior (DB). Furthermore, the stability of scores across two school subjects (i.e., German language, mathematics) was examined. A total of 20 students and two teachers in an inclusive elementary school classroom participated in the study. Generalizability study results suggest that little variance in DBR scores was attributable to the facets of raters or occasions, but the interactions between persons with raters and occasions accounted for a large portion of variance. Variance attributable to these interactions was slightly higher for DBR-MIS than for DBR-SIS ratings. Decision studies revealed that dependable measurements of AE could be achieved within 1.5 weeks of daily ratings for absolute decision-making. Findings regarding DB were mixed. Differences across both school subjects were found indicating that DBR ratings that are obtained in one academic setting may not necessarily be exchangeable with ratings obtained in another. Overall, the results demonstrate the utility of DBR for behavioral progress-monitoring purposes.

Concurrent validity and sensitivity to change of Direct Behavior Rating Single-Item Scales (DBR-SIS) within an elementary sample.

The purpose of this study was to evaluate the concurrent validity, sensitivity to change, and teacher acceptability of Direct Behavior Rating single-item scales (DBR-SIS), a brief progress monitoring measure designed to assess student behavioral change in response to intervention. Twenty-four elementary teacher-student dyads implemented a daily report card intervention to promote positive student behavior during prespecified classroom activities. During both baseline and intervention, teachers completed DBR-SIS ratings of 2 target behaviors (i.e., Academic Engagement, Disruptive Behavior) whereas research assistants collected systematic direct observation (SDO) data in relation to the same behaviors. Five change metrics (i.e., absolute change, percent of change from baseline, improvement rate difference, Tau-U, and standardized mean difference; Gresham, 2005) were calculated for both DBR-SIS and SDO data, yielding estimates of the change in student behavior in response to intervention. Mean DBR-SIS scores were predominantly moderately to highly correlated with SDO data within both baseline and intervention, demonstrating evidence of the former's concurrent validity. DBR-SIS change metrics were also significantly correlated with SDO change metrics for both Disruptive Behavior and Academic Engagement, yielding evidence of the former's sensitivity to change. In addition, teacher Usage Rating Profile-Assessment (URP-A) ratings indicated they found DBR-SIS to be acceptable and usable. Implications for practice, study limitations, and areas of future research are discussed. (PsycINFO Database Record

Examining the Concurrent Criterion-Related Validity of Direct Behavior Rating–Single Item Scales With Students With Social Competence Deficits

A line of research has supported the development and validation of Direct Behavior Rating–Single Item Scales (DBR-SIS) for use in progress monitoring. Yet, this research was largely conducted within the general education setting with typically developing children. It is unknown whether the tool may be defensibly used with students exhibiting more substantial concerns, including students with social competence difficulties. The purpose of this investigation was to examine the concurrent validity of DBR-SIS in a middle school sample of students exhibiting substantial social competence concerns ( n = 58). Students were assessed using both DBR-SIS and systematic direct observation (SDO) across three target behaviors. Each student was enrolled in one of two interventions: the Social Competence Intervention or a business-as-usual control condition. Students were assessed across three time points, including baseline, mid-intervention, and postintervention. A review of across-time correlations indicated small to moderate correlations between DBR-SIS and SDO data ( r = .25–.45). Results further suggested that the relationships between DBR-SIS and SDO targets were small to large at baseline. Correlations attenuated over time, though differences across time points were not statistically significant. This was with the exception of academic engagement correlations, which remained moderate–high across all time points.

Bridging the Gap: Direct Behavior Rating–Single Item Scales

Direct Behavior Ratings (DBRs) are behavioral assessment methods that combine the benefits of systematic direct observation and behavior rating scales. That is, DBRs involve the observation of operationally defined target behaviors during a prespecified observation period and the evaluation of those behaviors via brief ratings. In this way, DBR is considered a hybrid assessment method. This article describes the landscape of DBR tools and provides an overview of the procedures and research currently available to support the use of one specific version, DBR–Single Item Scales (DBR-SIS), for progress monitoring within multi-tiered systems of support.

Direct Behavior Ratings: A Feasible and Effective Progress Monitoring Approach for Social and Behavioral Interventions

Despite significant advancements in supporting students’ social, emotional, and behavioral health in recent years, progress remains hindered by limited formative assessment options. The focus of this special issue is on applications for Direct Behavior Rating (DBR), which is a hybrid assessment method incorporating procedures utilized in direct observations and teacher ratings. Three articles in the special issue utilize multiple baseline designs to demonstrate the utility and effectiveness of DBR single-item scales, and a fourth article describes a multiple-item DBR applied to social behaviors. Expert commentaries provide perspectives on the current literature related to the assessment of social, emotional, and behavioral constructs in school settings and also chart a future course for continued study.

Formative Assessment Using Direct Behavior Ratings: Evaluating Intervention Effects of Daily Behavior Report Cards

This study examined the treatment sensitivity of Direct Behavior Rating–Single Item Scales (DBR-SIS) in response to an evidence-based intervention delivered in a single-case, multiple-baseline design. DBR-SIS was used as a formative assessment in conjunction with a frequently used intervention in schools, a Daily Behavior Report Card (DRC). The intervention and concurrent assessment were conducted by five teachers in a rural Midwestern elementary school with five male students displaying mild to moderate behavioral challenges in the classroom. Study findings indicated DBR-SIS displays appropriate treatment sensitivity following intervention implementation. Agreement in the documentation of response and nonresponse to intervention implementation between DBR-SIS and systematic direct observation (SDO) data was evident across visual and empirical analyses. In addition, through a multiple-baseline design, this study documented negligible to no change in student behavior following implementation of a DRC in an applied classroom setting. These findings support previous calls for continued examination of the forms and components of DRC employed in schools. Finally, the study found educators rated the use of a combined DRC intervention and progress monitoring with DBR-SIS as favorable.

Progress Monitoring the Effects of Daily Report Cards Across Elementary and Secondary Settings Using Direct Behavior Rating: Single Item Scales

Direct Behavior Rating–Single Item Scales (DBR-SIS) have been advanced as a promising, systematic, behavioral, progress-monitoring method that is flexible, efficient, and defensible. This study aimed to extend existing literature on the use of DBR-SIS in elementary and secondary settings, and to examine methods of monitoring student progress in response to intervention. To this end, two concurrent multiple baseline design studies were conducted in a diverse magnet school district located in the northeastern United States. One study was conducted with four students in kindergarten and first grade, whereas the second study was conducted with three students in 10th and 11th grade. Response to a Daily Report Card (DRC) intervention was monitored using two different approaches: DBR-SIS and systematic direct observation (SDO) probes. Across all participants, modest improvements in behavior were observed using both visual and quantitative analyses of DBR-SIS data; however, decisions regarding student response to the intervention varied as a function of the dependent variable analyzed. Implications for practice and future directions for research are discussed.

Evaluating the technical adequacy of DBR-SIS in tri-annual behavioral screening: A multisite investigation

The implementation of multi-tiered systems in schools necessitates the use of screening assessments which produce valid and reliable data to identify students in need of tiered supports. Data derived from these screening assessments may be evaluated according to their classification accuracy, or the degree to which cut scores correctly identify individuals as “at-risk” or “not-at-risk.” The current study examined the performance of mean scores derived from over 1700 students in Grades 1, 2, 4, 5, 7, and 8 using Direct Behavior Rating—Single Item Scales. Students were rated across three time points (Fall, Winter, Spring) by their teachers in three areas: (a) academically engaged behavior, (b) disruptive behavior, and (c) respectful behavior. Classification accuracy indices and comparisons among behaviors were derived using Receiver Operating Characteristic (ROC) curve analyses, partial area under the curve (pAUC) tests, and bootstrapping methods to evaluate the degree to which mean behavior ratings accurately identified students who demonstrated elevated behavioral symptomology on the Behavioral and Emotional Screening System. Results indicated that optimal cut-scores for mean behavior ratings and a composite rating demonstrated high levels of specificity, sensitivity, and negative predictive value, with sensitivity point estimates for optimal cut-scores exceeding .70 for individual behaviors and .75 for composite scores across grade groups and time points.

The Impact of Target, Wording, and Duration on Rating Accuracy for Direct Behavior Rating

The purpose of this study was to extend evaluation of rater accuracy using Direct Behavior Rating–Single-Item Scales (DBR-SIS). Extension of prior research was accomplished through use of criterion ratings derived from both systematic direct observation and expert DBR-SIS scores, and also through control of the durations over which the target behavior was displayed in each clip. Rating targets included academically engaged, disruptive, and respectful behavior. Undergraduate participants ( n = 113) viewed video clips of classroom instruction and subsequently rated target students’ behavior using DBR-SIS. Results indicated that both rater bias and rater error were present to some degree based on the behavior target, wording, and duration displayed, with greater bias found for the target involving respectful behavior. In addition, positive wording for academically engaged behavior resulted in improved accuracy, yet negative wording resulted in improved accuracy for disruptive and possibly respectful behavior. Implications for research and practice are discussed.

Direct behavior rating as a school-based behavior screener for elementary and middle grades

The purpose of this study was to investigate how Direct Behavior Rating Single Item Scales (DBR-SIS) involving targets of academically engaged, disruptive, and respectful behaviors function in school-based screening assessment. Participants included 831 students in kindergarten through eighth grades who attended schools in the northeastern United States. Teachers provided behavior ratings for a sample of students in their classrooms on the DBR-SIS, the Behavioral and Emotional Screening System (Kamphaus & Reynolds, 2007), and the Student Risk Screening Scale (Drummond, 1994). Given variations in rating procedures to accommodate scheduling differences across grades, analysis was conducted separately for elementary school and middle school grade levels. Results suggested that the recommended cut scores, the combination of behavior targets, and the resulting conditional probability indices varied depending on grade level grouping (lower elementary, upper elementary, middle). For example, for the lower elementary grade level grouping, a combination of disruptive behavior (cut score=2) and academically engaged behavior (cut score=8) was considered to offer the best balance among indices of diagnostic accuracy, whereas a cut score of 1 for disruptive behavior and 8 for academically engaged behavior were recommended for the upper elementary school grade level grouping and cut scores of 1 and 9, respectively, were suggested for middle school grade level grouping. Generally, DBR-SIS cut scores considered optimal for screening using single or combined targets including academically engaged behavior and disruptive behavior by offering a reasonable balance of indices for sensitivity (.51–.90), specificity (.47–.83), negative predictive power (.94–.98), and positive predictive power (.14–.41). The single target of respectful behavior performed poorly across all grade level groups, and performance of DBR-SIS targets was relatively better in the elementary school than middle school grade level groups. Overall, results supported that disruptive behavior is highly important in evaluating risk status in lower grade levels and that academically engaged behavior becomes more pertinent as students reach higher grade levels. Limitations, future directions, and implications are discussed.

Evaluating Sensitivity to Behavioral Change Using Direct Behavior Rating Single-Item Scales

In this study, the researchers evaluated the sensitivity of Direct Behavior Rating Single-Item Scales (DBR-SIS) for assessing behavior change in response to an intervention. Included in the analyses were data from 20 completed behavioral consultation cases involving a diverse sample of elementary participants and contexts using a common intervention in an A-B design. Secondary purposes of the study were to investigate the utility of 5 metrics proposed for understanding behavioral response, as well as the correspondence among these metrics and teachers' ratings of intervention acceptability. Overall, results suggest that DBR-SIS demonstrated sensitivity to behavior change regardless of the metric used. Further, we found limited association between student change and teachers' ratings of acceptability.

The Influence of Alternative Scale Formats on the Generalizability of Data Obtained From Direct Behavior Rating Single-Item Scales (DBR-SIS)

The current study served to extend previous research on scaling construction of Direct Behavior Rating (DBR) in order to explore the potential flexibility of DBR to fit various intervention contexts. One hundred ninety-eight undergraduate students viewed the same classroom footage but rated student behavior using one of eight randomly assigned scales (i.e., differed with regard to number of gradients, length of scale, discrete vs. continuous). Descriptively, mean ratings typically fell within the same scale gradient across conditions. Furthermore, results of generalizability analyses revealed negligible variance attributable to the facet of scale type or interaction terms involving this facet. Implications for DBR scale construction within the context of intervention-related decision making are presented and discussed.

Preliminary evaluation of various training components on accuracy of Direct Behavior Ratings

This study examined the impact of various components of rater training on the accuracy of rating behavior using Direct Behavior Rating-Single Item Scales (DBR-SIS). Specifically, the addition of frame-of-reference and rater error training components to a standard package involving an overview and then modeling, practice, and feedback was investigated. In addition, amount of exposure to the direct training component (i.e., number of practice and feedback opportunities) was evaluated, and the rates at which behavior was displayed were carefully manipulated to control for and evaluate training impact by target and rate of behavior. The sample consisted of undergraduate students assigned to one of 6 possible conditions. Overall findings suggested that completion of a training package did result in enhanced accuracy when using DBR-SIS to rate academic engagement and disruption. However, results also supported that the most comprehensive package of DBR training may not always result in greater improvements over a standard package involving direct training. In general, a more intensive training package appeared beneficial at improving ratings for targets that had previously been difficult to rate accurately (e.g., medium rate disruptive behavior). Limitations and implications for future research are discussed.

An Evaluation of a Classwide Intervention Package Involving Self-Management and a Group Contingency on Classroom Behavior of Middle School Students

The effectiveness of an intervention package involving self-management and a group contingency at increasing appropriate classroom behaviors was evaluated in a sample of middle school students. Participants included all students in each of the 3 eighth-grade general education classrooms and their teachers. The intervention package included strategies recommended as part of best practice in classroom management to involve both building skill (self-management) and reinforcing appropriate behavior (group contingency). Data sources involved assessment of targeted behaviors using Direct Behavior Rating—single item scales completed by students and systematic direct observations completed by external observers. Outcomes suggested that, on average, student behavior moderately improved during intervention as compared to baseline when examining observational data for off-task behavior. Results for Direct Behavior Rating data were not as pronounced across all targets and classrooms in suggesting improvement for students. Limitations and future directions, along with implications for school-based practitioners working in middle school general education settings, are discussed.

An investigation of the generalizability and dependability of Direct Behavior Rating Single Item Scales (DBR-SIS) to measure academic engagement and disruptive behavior of middle school students

A total of 4 raters, including 2 teachers and 2 research assistants, used Direct Behavior Rating Single Item Scales (DBR-SIS) to measure the academic engagement and disruptive behavior of 7 middle school students across multiple occasions. Generalizability study results for the full model revealed modest to large magnitudes of variance associated with persons (students), occasions of measurement (day), and associated interactions. However, an unexpectedly low proportion of the variance in DBR data was attributable to the facet of rater, as well as a negligible variance component for the facet of rating occasion nested within day (10-min interval within a class period). Results of a reduced model and subsequent decision studies specific to individual rater and rater type (research assistant and teacher) suggested degree of reliability-like estimates differed substantially depending on rater. Overall, findings supported previous recommendations that in the absence of estimates of rater reliability and firm recommendations regarding rater training, ratings obtained from DBR-SIS, and subsequent analyses, be conducted within rater. Additionally, results suggested that when selecting a teacher rater, the person most likely to substantially interact with target students during the specified observation period may be the best choice.