Adequate Inter-rater Reliability Research Articles

Test-retest and inter-rater reliability and construct validity of the 2-minute step test in individuals with Parkinson's disease

ObjectiveTo evaluate the test-retest and inter-rater reliability and construct validity of the 2-minute step test (2MST) in individuals with Parkinson's disease (PD). MethodsTwo previously trained examiners applied the 2MST in two moments. Clinical measurements of PD were obtained using the following instruments: Berg Balance Scale (BBS), Unified Parkinson's Disease Rating Scale (UPDRS), modified Hoehn and Yahr Scale (H&Y), gait speed and cadence, 5-time Sit-to-Stand Test (5TSST) and Timed Up and Go (TUG). For reliability, the intraclass correlation coefficient (ICC), standard error of measurement (SEM) and minimal detectable change (MDC) were calculated. For construct validity, Pearson's and Spearman's correlation coefficient were used. The agreement between the measurement in person and via recorded video was assessed by means of Bland-Altman plots. ResultsThe 2MST showed adequate test-retest reliability (ICC = 0.80, SEM = 10.59, MDC = 29.35) and excellent inter-rater reliability (ICC = 0.93, SEM = 4.49, MDC = 12.46). There was acceptable agreement between assessments made with in person and via video. A correlation magnitude above 0.50 was found between 2MST and gait speed, and a correlation magnitude between 0.30 and 0.50 with BBS, gait cadence, UPDRS and TUG. ConclusionThe 2MST presents adequate test-retest and inter-rater reliability and a valid construct to evaluate the aerobic endurance of individuals with PD.

Assessing Participation in Stroke Survivors: Psychometric Properties of the Participation Measurement Scale (PM-Scale) Brazil.

Initial evidence supports the reliability of the Participation Measurement Scale (PM-Scale) in Brazil, but further exploration of its psychometric properties is needed for Brazilian stroke survivors. The aim of the study is to analyze the reliability, convergent validity, internal consistency, and accuracy of the PM-Scale Brazil. A methodological study involved three assessments over intervals of 7-14 days. Reliability was evaluated through the Intraclass Correlation Coefficient. Convergent validity was assessed using Spearman's correlation coefficient to evaluate the alignment of the PM-Scale Brazil with the SATIS-Stroke. Internal consistency was assessed through Cronbach's α. Accuracy was estimated by calculating the area under the receiver operating characteristic curve. The study involved 110 stroke survivors, revealing adequate intra-rater and inter-rater reliability. A significant weak correlation was observed between the PM-Scale and SATIS-Stroke. Internal consistency and accuracy were adequate. The PM-Scale Brazil demonstrates good reliability, internal consistency, and sensitivity. However, its weak correlation with SATIS-Stroke suggests limited convergent validity.

Real‐world evaluation of interconsensus agreement of risk of bias tools: A case study using risk of bias in nonrandomized studies‐of interventions (ROBINS‐I)

AbstractBackgroundRisk of bias (RoB) tools are critical in systematic reviews and affect subsequent decision‐making. RoB tools should have adequate interrater reliability and interconsensus agreement. We present an approach of post hoc evaluation of RoB tools using duplicated studies that overlap systematic reviews.MethodsUsing a back‐citation approach, we identified systematic reviews that used the Risk Of Bias In Nonrandomized Studies‐of Interventions (ROBINS‐I) tool and retrieved all the included primary studies. We selected studies that were appraised by more than one systematic review and calculated observed agreement and unweighted kappa comparing the different systematic reviews' assessments.ResultsWe identified 903 systematic reviews that used the tool with 51,676 cited references, from which we eventually analyzed 171 duplicated studies assessed using ROBINS‐I by different systematic reviewers. The observed agreement on ROBINS‐I domains ranged from 54.9% (missing data domain) to 70.3% (deviations from intended interventions domain), and was 63.0% for overall RoB assessment of the study. Kappa coefficient ranged from 0.131 (measurement of outcome domain) to 0.396 (domains of confounding and deviations from intended interventions), and was 0.404 for overall RoB assessment of the study.ConclusionA post hoc evaluation of RoB tools is feasible by focusing on duplicated studies that overlap systematic review. ROBINS‐I assessments demonstrated considerable variation in interconsensus agreement among various systematic reviewes that assessed the same study and outcome, suggesting the need for more intensive upfront work to calibrate systematic reviewers on how to identify context‐specific information and agree on how to judge it.

Development and Content and Face Validation of Low-Cost Simulators Evaluation Instrument

BackgroundDeveloping low-cost simulators is an evolving area within clinical simulation. As such, there is a need for instruments to assess these simulators. The study aimed to develop and evaluate an instrument's face and content validity to assess if an own-made low-cost simulator meets the essential characteristics low-cost simulators should have. MethodMethodological study in two phases. Phase 1 developed the instrument's items, and phase 2 assessed the content and face validity of the instrument and items developed in the previous phase. Twenty-six experts evaluated the face and content validation of the instrument in terms of its representativeness, relevance, clarity and inter-rate reliability. ResultsTwenty-five items distributed in six factors were proposed to experts. All factors and items showed adequate content validity indexes and inter-rater reliability. Face validity was also endorsed for the instrument to represent the desirable key attributes of low-cost simulators. ConclusionThis study provides The Low-Cost Simulator Assessment Instrument, available in Spanish and Portuguese. The instrument, composed of twenty-five items distributed in six factors, has shown adequate content and face validity.

Validation of the Spanish version of the Pediatric Asthma Severity Score (PASS) in a population of Hispanic children.

The Pediatric Asthma Severity Score (PASS)is one of the most-used clinical scoring systems for assessing the severity of asthma exacerbations in children. The aim of the present study was to validate a Spanish version of the PASS in a population of Hispanic children with asthma exacerbations living in urban Bogota, Colombia. In a prospective cohort and a validation study, parents/caregivers of children between 2 and 18 years old attended in the emergency department (ED)with asthma exacerbations who were admitted to the inpatient unit were invited to participate in the study. During the hospitalization period, we gathered the necessary data for assessing the criterion validity (comparing its score with the Pediatric Respiratory Assessment Measure [PRAM]), construct validity, interrater reliability, responsiveness, and internal consistency of the Col-PASS,the Colombian version of the PASS. At baseline, the scores of the Col-PASScorrelated positively with the scores of the PRAMscore (ρ = 0.588, p < .001). The baseline Col-PASSscores in patients who required admission to a more complex service were significantly higher than those in patients who presented clinical improvement (1.0 (0.0-2.0) vs. 0.0 (0.0-0.0), p < .001). The interrater reliability was found to be κ = 0.897, 95% CI 0.699-1.000, p < .001. Cronbach's α was .701 for the questionnaire as a whole. The Col-PASShas excellent construct validity, adequate criterion validity, interrater reliability, responsiveness; and acceptable internal consistency when used in children between 2 and 18 years old with asthma exacerbations.

Characterizing Interventions Used to Promote Life Participation in Adults on Peritoneal Dialysis Therapy: A Scoping Review.

Living with kidney failure can interfere with life participation (ie, participation in valued life activities). Life participation has recently been identified as a top-priority health outcome of people on peritoneal dialysis therapy, but it is a relatively unexplored topic in peritoneal dialysis. The objective is to describe the interventions that have been used to promote life participation in the peritoneal dialysis population and highlight research gaps warranting further investigation. A scoping review was conducted according to the Joanna Briggs Institute methodology. Six electronic databases (MEDLINE [OVID], EMBASE, PsycINFO, Cochrane Central Register of Controlled Trials, CINAHL Plus, SCOPUS) were searched. Adults aged 18+ years on peritoneal dialysis therapy. Any dedicated scale or subscale that measured life participation as an isolated outcome. Title/abstract screening was completed independently after adequate inter-rater reliability (kappa > 0.8) was achieved among reviewers. Full-text review and data extraction were conducted in duplicate. Extracted data were analyzed using counts, percentages, and narrative synthesis to describe patterns in the literature. After identifying 13 874 results, 17 studies met eligibility criteria. Eight studies were conducted within the past 5 years, with China as the most common study location. Only 2 studies investigated life participation as a primary study outcome. Eight studies targeted personal-physical barriers to life participation, 8 targeted multiple barriers, and 1 targeted an environmental-institutional barrier. Life participation was assessed within a subdomain of a broader quality of life assessment (The Kidney Disease Quality of Life [KDQOL]-36 or the 36-Item Short-Form Health Survey [SF-36]) in 11 studies. The majority of assessments captured life participation in all major domains of participation (self-care, work, and leisure). Eligibility screening at title/abstract stage was not performed in duplicate; articles not available in English were excluded. Life participation has infrequently been prioritized as a health outcome in peritoneal dialysis (PD). Interventions have been narrow in focus given the range of challenges faced by people on PD and the holistic approaches used in other clinical populations. Future research should prioritize life participation as a key health outcome in PD and investigate the impact of interventions that address cognitive, affective, and environmental barriers to participation.

Disease-Modifying Therapies for Sickle Cell Disease: Decisional Needs and Supports Among Adolescents and Young Adults

Introduction: Adolescents and young adults (AYAs) with sickle cell disease (SCD) often experience significant challenges in disease management. Hydroxyurea and several newer disease-modifying therapies have demonstrated health benefits; however, uptake of these therapies remains suboptimal. Shared decision-making is one promising solution to addressing barriers in uptake and to improve health outcomes for AYAs with SCD. A thorough understanding of AYA's decisional needs can guide the development of tailored decisional supports and promote shared decision-making. Informed by the Ottawa Decision Support Framework, this study aimed to examine decisional needs and supports reported by AYAs with SCD, their caregivers, and SCD healthcare providers. Methods: Semi-structured qualitative interviews were conducted with AYAs ages 15-25 years with SCD ( n=14) and caregivers of AYAs with SCD ( n=11) receiving care at two hospitals in the mid-Atlantic region of the US. Qualitative data were also obtained through online crowdsourcing with SCD providers across the US ( n=40). Most AYAs (93%) identified as non-Hispanic Black ( M age=21 years, 57% male) and had SS genotype (79%). Most caregivers were female (80%) and all identified as non-Hispanic Black (100%). About two-thirds of healthcare providers were female (65%) and non-Hispanic White (65.0%), with an average of 14.8 years in practice ( range=2-57 years). Thematic and descriptive content analyses were used to summarize perspectives on decisional needs and supports regarding disease-modifying therapies. Adequate inter-rater reliability (Cohen's Kappas &amp;gt; .80) and thematic saturation were achieved. Results: AYAs and caregivers reported needing six areas of support when deciding on disease-modifying therapies: resolving decisional conflicts, gaining knowledge, clarifying expectations, garnering supports and resources, navigating complexities in the decision, and addressing other personal concerns. Families reported receiving supports from providers that facilitated their decision-making by helping them: gain knowledge, clarify expectations, address other personal concerns, build rapport with providers, and resolve unreceptiveness or ambivalence to making a decision. Providers reported offering additional supports targeting: inadequate experience in implementing therapies (e.g., pill swallowing), inadequate health and social services and financial assistance, and inadequate instrumental support (e.g., transportation). Conclusions: This research highlights the needs AYAs and caregivers identify when making decisions about disease-modifying therapies for SCD and the decision supports offered by healthcare providers. Decisional needs reported by families generally corresponded to supports offered by providers. Providers also highlighted additional supports that may be helpful for families (e.g., instrumental supports, such as financial, emotional, and skillset assistance). Findings will inform the development of a shared decision-making intervention for AYAs with SCD, their caregivers, and healthcare providers. Future research is needed to examine which decision supports may be most impactful to promote shared decision-making for disease-modifying therapies.

The development and preliminary evaluation of the Genetic Counseling Skills Checklist.

Genetic counseling (GC) relies on communication to help people understand and adapt to genetic contributions of disease, and there is need for a practical and reliable method of comprehensively documenting GC communication skills without intensive coding. To this end, we created a novel process measure called the Genetic Counseling Skills Checklist (GCSC), utilizing previously validated measures, communication/counseling frameworks, and prior research findings. A multistage iterative process was used to develop, evaluate, and modify the GCSC to improve its clarity, usability, and content validity. To assess interrater reliability, randomly assigned, untrained individuals (i.e., coders) used the GCSC version 3 to code multiple simulated GC sessions. Average measures intraclass correlation coefficients (ICCs) were calculated for each of the 8 GCSC process categories using one-way, random effects models. After relatively minor modifications to the GCSC, two pairs of experienced coders used GCSC version 4 to independently code additional GC sessions and Cohen's Kappa coefficients (κ) were calculated to assess interrater reliability for each process category. The GCSC contains five to eight items within each category and demonstrates good content validity given its ability to capture nearly all GC skills that genetic counselors reported using in a prior qualitative study. Interrater reliability of GCSC version 3 among coders with limited experience was moderate or good for 6 out of the 8 process categories as evidenced by ICCs ranging from 0.55 to 0.86. Average interrater reliability of GCSC version 4 among one pair of experienced coders was strong for all eight process categories (κ ranging from 0.82 to 0.94); among the second pair of experienced coders scores were strong for six categories (κ ranging from 0.80 to 0.87) and moderate for two categories (κ of 0.77 and 0.78). The results suggest the need for training and experience to assure adequate interrater reliability across GCSC coders. Future work is needed to create a formalized training program for coders, complete a larger study to further validate the measure, and use the GCSC to document variability in skills used across providers and sessions.

Testing of the Facial Expression Coding System (FACES) for middle school–aged special education students and development of a training protocol

Emotion regulation is a key developmental skillset, but many existing measures rely on self-report or laboratory-based measurement approaches. This study aimed to develop a training and implementation protocol for the widely used Facial Expression Coding System (FACES) to be used in real-world settings with pre-recorded video data. A revised coding system with supplemental guidelines and training procedures was developed to use FACES with video data recorded in special education classrooms. This system resulted in adequate interrater reliability as well as reduced training time for coders. Specific training methods included close study of code definitions, coding of practice video, quantitative analysis of observation data to generate interrater agreement and kappa statistics, review of comparison charts to identify discrepancies between coder and training observations using the Noldus Observer XT software, and post-observation discussions. The revised FACES protocol and new training method presented here offer a more robust, efficient, and versatile tool that can be applied to systematic behavior observations conducted of students in real-world classroom settings.

Neighborhood conditions in a Swedish context-Two studies of reliability and validity of virtual systematic social observation using Google Street View.

The goal of these studies was to investigate the reliability and validity of virtual systematic social observation (virtual SSO) using Google Street View in a Swedish neighborhood context. This was accomplished in two studies. Study 1 focused on interrater reliability and construct validity, comparing ratings conducted in-person to those done using Google Street View, across 24 study sites within four postal code areas. Study 2 focused on criterion validity of virtual SSO in terms of neighborhoods with low versus high income levels, including 133 study sites within 22 postal code areas in a large Swedish city. In both studies, assessment of the neighborhood context was conducted at each study site, using a protocol adapted to a Swedish context. Scales for Physical Decay, Neighborhood Dangerousness, and Physical Disorder were found to be reliable, with adequate interrater reliability, high consistency across methods, and high internal consistency. In Study 2, significantly higher levels of observed Physical Decay, Neighborhood Dangerousness, and signs of garbage or litter were observed in postal codes areas (site data was aggregated to postal code level) with lower as compared to higher income levels. We concluded that the scales within the virtual SSO with Google Street View protocol that were developed in this series of studies represents a reliable and valid measure of several key neighborhood contextual features. Implications for understanding the complex person-context interactions central to many theories of positive development among youth were discussed in relation to the study findings.

Psychometric properties of the Observing Mediational Interactions (OMI) coding system during mother-adolescent conflict discussions.

Observational assessments of parent-adolescent conflict can guide interventions to prevent and reduce conflict and mental health problems. The authors identified the Observing Mediational Interactions (OMI) as a particularly useful coding system for examining parent-adolescent conflict. The OMI is the observational measure used in the Mediational Intervention for Sensitizing Caregivers (MISC) and quantifies emotional (attachment-based) and cognitive (learning-based) behaviors during caregiver-child interactions. The overall aim of the current study was to tailor and evaluate the OMI specifically for observing conflict interactions. Conflict discussions between 56 clinical youth aged 10-15 years and their mothers were coded using the OMI. Reliability, construct validity, and associations with affect following the conflict discussion were examined. Analyses revealed that the OMI demonstrated adequate internal consistency, interrater reliability, and construct validity in terms of associations with an alternate observational measure and parent-reported family functioning and stress. Additionally, mothers who engaged in fewer negative parenting behaviors reported greater positive affect following the conflict discussion, controlling for baseline affect. Taken together, findings support the use of the OMI in future studies. While further research should attempt to extend findings to other populations and settings and elaborate the coding instructions as needed, researchers can use the OMI to inform ongoing adaptations of MISC and identify targets for prevention and intervention more broadly.

The Performance of the Health Communication Assessment Tool© (HCAT-f) in Calibrating Different Levels of Nurse Communication Skills in a French-Speaking Context

Communication skills training is essential in nurse education. Miscommunication may lead to adverse events and unsafe healthcare. To date, valid and reliable instruments to serve both communication training and assessment purposes across different cultural contexts are scarce. The present study empirically tested a French-language version of the Health Communication Assessment Tool© (HCAT-f) across different levels of communication skills performance to establish its reliability and validity through a cognitive fluency framework. Ten experts in communication and 52 nurse educators rated three videos simulating conversations between a nurse and a patient scheduled for lumpectomy. Each video captured a different level of communication skills performed by the nurse: High, medium, and low. Three distinct constructs were identified, i.e., professional presentation, empathy, and trust building. At absolute single-measure, an ICC = .43 suggested adequate interrater reliability of the whole scale for the medium-performed scenario, which decreased in low-performed (ICC = .35) and high-performed (ICC = .18) scenarios. The HCAT-f fulfils the criteria of linguistic equivalence, contextual relevance, and demonstrates acceptable construct validity. It can be used as a summative assessment tool after prior training on scale calibration is in place because interrater agreement was difficult to be established in high and low performance scenarios.

Measuring respiratory syncytial virus infection severity in hospitalized children using the Pediatric Respiratory Syncytial Virus Electronic Severity and Outcome Rating System (PRESORS)

The Pediatric Respiratory Syncytial Virus Electronic Severity and Outcome Rating System (PRESORS) was developed to assess the severity of respiratory syncytial virus (RSV) infection in children. Because young children cannot report how they feel or function, ratings are based on observations by the child's caregiver (Observer-Reported Outcome questionnaire [ObsRO]) and clinician (Clinician-Reported Outcome questionnaire [ClinRO]). This prospective study aimed to evaluate the psychometric properties of the PRESORS. The PRESORS version 6 ObsRO and ClinRO were evaluated in children with RSV infection requiring hospitalization in centers in the United States, Argentina, and Chile. Assessments were performed from days 1 to 7 by the child's caregiver and clinician. To assess inter-rater reliability, two clinicians independently performed the ClinRO near in time. A total of 124 children aged ≤36 months were enrolled (mean age, 8months). Factor analysis demonstrated that RSV severity consists of two dimensions, respiratory signs and illness behavior, and that these dimensions were consistent over time. The inter-rater reliability for the ClinRO was 0.66 (95% confidence interval [CI], 0.55-0.75) but improved to 0.79 (95% CI, 0.71-0.86) after removing one outlying site, suggesting that quantifying RSV severity is not trivial, even using qualified raters, but that an adequate inter-rater reliability is achievable with the PRESORS through adequate training. ClinRO and ObsRO displayed acceptable internal consistency and acceptable convergent validity with the Respiratory Syncytial Virus Network Scale, global impression scores, and key hospital characteristics. The PRESORS is relevant and appropriate for assessing the severity of RSV infection in infants requiring hospitalization.

How Much Training Do We Need? Assessing the Validity and Interrater Reliability of the PDM-2’s Psychodiagnostic Chart among Less Experienced Clinicians

The Psychodynamic Diagnostic Manual – Second Edition (PDM-2) has emerging evidence supporting its clinical utility, yet one of the main limitations remains the conjecture that considerable training and experience is required. It also remains unclear how the PDM-2 framework compares with current DSM measures of personality pathology such as the Level of Personality Functioning Scale (LPFS). The aim of the present study is to examine these issues by testing whether less clinically experienced second year doctoral psychology students are able to reliably assess patients’ level of personality pathology using the PDM-2’s Psychodiagnostic Chart (PDC-2), and to investigate the convergence between the PDC-2 and the LPFS. Results showed adequate inter-rater reliability for both of the main PDC-2 axes, with 52% of the variance for the overall personality organization (P-Axis) rating, and 29% of the overall M-Axis score being due to rater consensus. Reliability of individual ratings ranged from fair to excellent for the overall scores on both axes (ICC = 0.59 to .90). Results also showed that student evaluations were valid, with the latter’s assessment of the patients’ level of personality organization converging both with the experts’ rating of the PDC-2 as well as the LPFS. Implications for clinical training are discussed.

Facilitative interpersonal skills are relevant in child therapy too, so why don't we measure them?

One of the consistent findings from psychotherapy process research has been the impact of therapist effects on patient change and the therapeutic alliance. The Facilitative Interpersonal Skills (FIS) paradigm is a task in which participants respond to standardized videos of actors playing patients in interpersonally challenging moments as if they were the therapist, which was designed to assess therapist effects. Participants’ video recorded responses are coded for eight skills: verbal fluency, emotional expressiveness, warmth/acceptance/understanding, empathy, persuasiveness, hope/positive expectations, alliance-bond capacity, and rupture-repair responsiveness. Performance-based procedures like the FIS minimize self-report bias and systematically control for client-related variability while maintaining strong clinical relevance. Research has shown that therapist FIS predicts the quality of the therapeutic alliance and outcome in adult psychotherapy. This paper describes the development and first adaptation of the FIS task using child and adolescent patients as the stimuli, and reports findings from a pilot study testing the reliability. The FIS-Child (FIS-C) task was administered to 10 therapists with a range of clinical backgrounds. Participants also completed the original FIS task and self-report measures of their empathy, social skills, and playfulness. Adequate interrater reliability was achieved on the FIS-C. There were no significant differences between participants’ ratings on the FIS-C compared to the original FIS, although there were minor differences in the correlations between the FISC and self-report measures compared to the original FIS. Findings support moving forward with utilizing the FIS-C to empirically study therapist effects that may be common factors across treatment models.

Psychometric Properties of The Clinician Affective Reactivity Index for Assessment of Irritability in a Clinical Sample of Turkish Children and Adolescents.

BackgroundNo clinician-oriented scale exists to assess irritability in Turkey. This pilot study aimed to evaluate the psychometric properties of the Turkish version of The Clinician Affective Reactivity Index (CL-ARI).MethodA total of 116 children and adolescents aged between 10 to 17 years (14.1 ± 2.1 years) were recruited from the psychiatric outpatient clinics. The participants completed a set of scales (Strengths and Difficulties Questionnaire [SDQ], Affective Reactivity Index [ARI], Revised Child Anxiety and Depression Scale, Swanson, Nolan, and Pelham, Version IV Scale). Diagnostic interviews were administered to confirm psychiatric diagnoses. Cronbach’s alpha was calculated to assess internal consistency. Discriminant validity was further tested using independent sample t-test and Receiver Operating Characteristic curves. Interrater reliability was tested using intraclass correlation coefficients (ICC). Convergent validity was also tested using Pearson’s correlation.ResultsCronbach’s alpha values of CL-ARI were 0.919 total score, 0.842 for the temper outbursts score, 0.861 for the irritable mood score, and 0.840 for the impairment score. ICC values for interrater reliability were high for the temper outbursts (r = 0.993), the irritable mood (r = 0.993), the impairment (r = 0.917), and the total score (r = 0.991). In the sample, there was a high level of correlation between the self-report ARI-child/parent form and the CL-ARI total and subscale scores. Likewise, moderate-high level of correlations were found between the behavioral SDQ child/parent forms and the CL-ARI total and subscale scores.ConclusionsThis is the Turkish validation of the CL-ARI, a dedicated interview and rating scale to assess irritability in the clinical sample. The results of this study suggest that the Turkish version of CL-ARI has adequate internal consistency and interrater reliability, and sufficient convergent and discriminant validity to be used in research settings.

SBAR-LA: SBAR Brief Assessment Rubric for Learner Assessment.

IntroductionStructured communication tools are associated with improvement in information transfer and lead to improved patient safety. Situation, Background, Assessment, Recommendation (SBAR) is one such tool. Because there is a paucity of instruments to measure SBAR effectiveness, we developed and validated an assessment tool for use with prepractice health professions students.MethodsWe developed the SBAR Brief Assessment Rubric for Learner Assessment (SBAR-LA) by starting with a preliminary list of items based on the SBAR framework. During an interprofessional team training event, students were trained in the use of SBAR. Subsequently, they were assigned to perform a simulated communication scenario demonstrating use of SBAR principles. We used 10 videos from these scenarios to refine the items and scales over two rounds. Finally, we applied the instrument on another subset of 10 students to conduct rater calibration and measure interrater reliability.ResultsWe used a total of 20 out of 225 videos of student performance to create the 10-item instrument. Interrater reliability was .672, and for eight items, the Fleiss’ kappa was considered good or fair.DiscussionWe developed a scoring rubric for teaching SBAR communication that met criteria for validity and demonstrated adequate interrater reliability. Our development process provided evidence of validity for the content, construct, and response process used. Additional evidence from the use of SBAR-LA in settings where communication skills can be directly observed, such as simulation and clinical environments, may further enhance the instrument's accuracy. The SBAR-LA is a valid and reliable instrument to assess student performance.

Testing Children\u2019s Mentalizing in Middle Childhood: Adopting the Child and Adolescent Reflective Functioning Scale with Clinical and Community Children

Beginning with Ensink’s seminal study (2015), the field entered a new era in which we were able to measure mentalizing in school-aged children. The goal of this work is to continue developing the state of the research within this tradition by exploring the psychometric properties of the Child Reflective Functioning Scale (CRFS) - a measure applied to the Child Attachment Interview and designed to assess RF during middle childhood - within both clinical and normative groups, and to examine if it differentiates between both groups. Participants were 159 Italian children (age range 8–12 years, Mage = 10.66, SD =1.83; 57% males) divided into two groups: 71 children with emotional-behavioral problems (‘clinical group’) and 88 children without emotional-behavioral problems (‘community group’). Demographic data and Verbal Comprehension Index were collected. A two-factor model of CRFS (self- and other-focused RF) was confirmed in both groups, revealing that these intrapersonal and interpersonal indicators of children’s RF are important domains of mentalizing abilities in middle childhood. The results revealed adequate inter-rater reliability of the CRFS. Controlling for the effect of potentially confounding demographic variables, significant differences on CRFS scales between clinical and community groups were found. The clinical group showed lower levels of CRFS scores than normative group, but importantly, the self-focused RF score uniquely predicted clinical/community status. Taken together, the findings showed that the CRFS is a reliable and validity measure for assessing RF in middle childhood with clinical and normative groups, contributing important information to the scientific literature on mentalizing in middle childhood.

A reliable interictal EEG grading scale for children with infantile spasms – The 2021 BASED score

ObjectiveTo develop an improved interictal electroencephalogram (EEG) grading scale for children with infantile spasms founded on elements with adequate inter-rater reliability (IRR) to justify its further study for clinical and research purposes. MethodsThree blinded reviewers assessed five-minute sleep epochs in 93 EEGs from 62 children (31 consecutive controls, 31 consecutive infantile spasms [pretreatment and posttreatment studies]) using a longitudinal bipolar montage. We determined the IRR of background amplitude, epileptiform discharges, >3 spike foci (including <50 % or >50 %), grouped multifocal spikes, paroxysmal voltage attenuations, and symmetry of sleep spindles. Data were used to finalize the 2021 BASED (Burden of AmplitudeS and Epileptiform Discharges) score. ResultsAll elements included in the 2021 BASED score had moderate to near perfect IRR. Among controls, >200 μv background waves occurred commonly in the bilateral posterior temporal (T3-T5, T4-T6) and midline (Fz-Cz, Cz-Pz) regions. Excluding midline and occipital channels (which have normal high amplitude background waves), we designated abnormal high amplitude background waves as >200 μv for most channels, but >300 μv for T3-T5 and T4-T6. The IRR was moderate to near perfect for <50 % >3 spike foci, >50 % >3 spike foci, paroxysmal voltage attenuations, grouped multifocal spikes (GMFS), and symmetric sleep spindles. Paroxysmal voltage attenuations, GMFS, and >50 % >3 spike foci all significantly distinguished pretreatment from posttreatment studies whereas symmetric sleep spindles did not (as planned, the latter was not included in the 2021 BASED score). When the 2021 BASED score was applied to the 22 children with infantile spasms achieving clinical remission with treatment, 19 met criteria for electroclinical remission and three did not. SignificanceThe 2021 BASED score includes elements with high levels of IRR and correlates well with the presence or absence of infantile spasms.

Improving Self-Reflection Assessment Practices: Comparative Judgment as an Alternative to Rubrics

Construct: The authors aimed to investigate the utility of the comparative judgment method for assessing students’ written self-reflections. Background: Medical practitioners’ reflective skills are increasingly considered important and therefore included in the medical education curriculum. However, assessing students’ reflective skills using rubrics does not appear to guarantee adequate inter-rater reliabilities. Recently, comparative judgment was introduced as a new method to evaluate performance assessments. This study investigates the merits and limitations of the comparative judgment method for assessing students’ written self-reflections. More specifically, it examines the reliability in relation to the time spent assessing, the correlation between the scores obtained using the two methods (rubrics and comparative judgment), and, raters’ perceptions of the comparative judgment method. Approach: Twenty-two self-reflections, that had previously been scored using a rubric, were assessed by a group of eight raters using comparative judgment. Two hundred comparisons were completed and a rank order was calculated. Raters’ impressions were investigated using a focus group. Findings: Using comparative judgment, each self-reflection needed to be compared seven times with another self-reflection to reach a scale separation reliability of .55. The inter-rater reliability of rating (ICC, (1, k)) using rubrics was .56. The time investment required for these reliability levels in both methods was around 24 minutes. The Kendall’s tau rank correlation indicated a strong correlation between the scores obtained via both methods. Raters reported that making comparisons made them evaluate the quality of self-reflections in a more nuanced way. Time investment was, however, considered heavy, especially for the first comparisons. Although raters appreciated that they did not have to assign a grade to each self-reflection, the fact that the method does not automatically lead to a grade or feedback was considered a downside. Conclusions First evidence was provided for the comparative judgment method as an alternative to using rubrics for assessing students’ written self-reflections. Before comparative judgment can be implemented for summative assessment, more research is needed on the time investment required to ensure no contradictory feedback is given back to students. Moreover, as the comparative judgment method requires an additional standard setting exercise to obtain grades, more research is warranted on the merits and limitations of this method when a pass/fail approach is used.