High Inter-rater Reliability Research Articles

Coaching in action: investigating actual coaching behaviour over time

ABSTRACT Research on the factors contributing to effective coaching has gained significant attention in recent years. These factors, however, are rather abstract constructs that cannot readily be observed. To address this issue, we focus on coaching behaviours. Coaching behaviours describe what coaches actually do during the coaching process and represent the visible and observable part of the coaching factors. Using behavioural observation, we analysed 50 coaching sessions with an average duration of 73.02 minutes (SD = 16.26). Our analysis demonstrated that specific coaching behaviours and coaching phases can be observed with high inter-rater reliability. In addition, coaching behaviours vary across different coaching phases, suggesting that coaches implement coaching factors in specific phases. These findings deepen our understanding of how coaching factors manifest themselves on a behavioural level throughout the coaching process, informing both research and evidence-based practice. This knowledge can also serve as a foundation for developing training programmes that target coaching processes at a fine-grained level.

Optimalisatie en validatie van de Vlaamse Observationele Pijnschaal bij Meervoudige Beperkingen (VOPS-MB) voor personen met een Meervoudige Beperking: een multidisciplinaire benadering

Optimization and validation of the Flemish Observational Pain Scale for Multiple Disabilities (VOPS-MB) for individuals with multiple disabilities: a multidisciplinary approach Introduction: People with severe cognitive impairments and multiple disabilities often experience pain, but this pain frequently goes unrecognized or untreated, negatively impacting their quality of life. Pain management in this group is challenging due to limited communication abilities and the lack of validated pain assessment tools. Aim: This study aims to optimize and validate the VOPS-MB pain scale of the Leieborg Care Center in Deinze for use with individuals with multiple disabilities. Methods: Optimization was carried out through a systematic literature review, an expert focus group, and a Delphi study. The scale's validation was tested through observations of 50 care recipients at the center, with inter-rater reliability measured using Pearson's r correlation coefficient and Cohen's kappa. Results: The optimized pain scale demonstrated high inter-rater reliability, with a Pearson's r of 0.947 and a Kappa value of 0.867. These results indicate that the scale provides reliable results, even when observations are conducted by individuals unfamiliar with the care recipient. Conclusion: This study shows that the pain scale is useful, practical and adequate to use in practice. Furthermore, the findings suggest that this pain scale can also be used in other contexts. The scale enables accurate pain measurement in people with intellectual disabilities, regardless of the observer's knowledge of the patient. Further validation, including digitization and testing for responsiveness, is recommended to enhance the scale's usability.

The Childhood Autism Rating Scale Second Edition (CARS2) and Its Applicability in an Iranian Sample.

The present study aimed to evaluate the reliability and validity of the Childhood Autism Rating Scale: Second Edition (CARS2) in diagnosing individuals with autism in Iran. A mixed-method approach was used and 313 participants were recruited, with an age range of 2-32 years, for CARS2-Standard Form (ST) and 218 individuals aged 6-25 years for CARS2-High Functioning (HF). The participants were recruited from daycare centers, schools, and clinics with different developmental trajectories: autism, intellectual disabilities, and neurotypical development. All participants with autism and intellectual disabilities had been clinically diagnosed previously. In addition, the CARS2-Questionnaire of Parent Concerns (QPC) was used to gather qualitative data on 30 randomly selected parents and the perspectives of the 20 test administrators were also collected. The CARS2 had high internal consistency, inter-rater reliability, and test-retest reliability. Factor analyses revealed a one-factor structure for CARS2-ST and a three-factor structure for CARS2-HF. When adjustments were made to the cut-off points, the discriminant analyses indicated that CARS2 effectively differentiated between those with autism and typical development but less so with persons who had intellectual disabilities. The qualitative data analysis and the extracted themes suggest that the CARS2-QPC is a valid tool for collecting autism-related information from parents. Our findings suggest that the CARS2 is broadly a reliable and valid instrument for diagnosing autism spectrum in Iran in the absence of more extensive assessments.

Screening Cognition, Sleep, and Physical Activity in Pediatric Oncology Long-Term Follow-Up Care.

Children treated for pediatric cancer are at risk for cognitive late effects, as well as impairments in sleep and physical activity. The aim of the present study was to examine the psychometric properties of clinical screening of child- and caregiver-reported cognitive functioning, sleep, and physical activity and the relationship between cognitive functioning and health behaviors within a pediatric oncology long-term follow-up clinic. The study included a retrospective chart review of 99 caregivers and 80 children (8-17years old) who completed the Conners Short Form (parent only) and PROMIS cognitive functioning, sleep-related impairment, sleep disturbance, and physical activity scales at the child's annual long-term follow-up visit. Test statistics and T-tests were used to assess psychometrics of the PROMIS scales. Bivariate correlations were used to examine the relationship between cognitive function and health behaviors. The child- and parent-report, short-version PROMIS cognitive functioning, sleep, and physical activity scales demonstrated high internal consistency and inter-rater reliability. High convergent validity was observed between PROMIS cognitive functioning and Conners Short Form. Caregiver- and child-reported cognitive functioning and health behaviors were significantly related (p ≤ 0.042). The PROMIS short-version scales are reliable and valid measures for screening cognitive function and health behaviors in pediatric oncology long-term follow-up care. Further research examining the predictive validity and longitudinal utility of the PROMIS scales in survivors is warranted. There was a positive association between cognitive functioning and health behaviors in survivors, warranting further investigation to inform potential targets of intervention in long-term follow-up care.

Effect of a Training System Utilizing 3D-Printed Patient-Specific Vascular Models on Endovascular Catheterization Performance.

This study aimed to examine the effect of simulation training using a three-dimensional (3D)-printed patient-specific vascular model on the advanced vascular catheterization skills of experienced interventional radiologists (IRs). Two specific anatomic types of 3D-printed patient-specific models from two patients with challenging celiac axis arterial anatomy were constructed. The Global Rating Scale of Endovascular Performance (GRS-EP) was used to evaluate vascular insertion skills. The training sessions comprised a pre- and post- training evaluation. Two blinded raters evaluated the effectiveness of the training. Improvements were observed in success rate, insertion time, and GRS-EP scores among all five experienced IRs. The GRS-EP demonstrated high inter-rater reliability. Post-training scores increased significantly in both video and self-evaluations. Although there is no proof that better performance on this type of model directly translates to improved performance in humans, simulation training using this model has the potential to help experienced IRs further refine their vascular catheterization skills.

Adaption and validation of the Greek version of Addenbrooke's Cognitive Examination III scale as a screening tool for perioperative cognitive impairment detection.

Postoperative cognitive dysfunction constitutes an extremely prevalent implication in individuals subjected to cardiac or non-cardiac surgery. This study aims to assess the validity and reliability of a culturally adapted Greek version of the Addenbrooke's Cognitive Examination III (ACE-III) scale as a screening tool for perioperative neurocognitive status determination in elderly surgical patients. A cross-cultural adaptation and validation of instruments throughout the a cross-sectional study was conducted. The study sample consisted of 128 individuals over 55 years old scheduled for surgical intervention. All participants were screened twice; at the preadmission clinic and the day before surgery using the already established MoCA scale and the culturally adapted Greek version of the ACE-III scale. Subjects with a MoCA score of < 26 constituted the cognitively impaired group. Regarding construct validity, ACE-III's performance in detecting cognitive impairment was excellent (AUC =0.942; 95%CI 0.899-0.971). Convergent validity between ACE-III and MoCA scales was excellent (r= 0.876; 95%CI 0.839-0.905). Known group validity was confirmed since advanced age and lower educational attainment adversely impacted ACE-III's total score (p<0.001). Additionally, specialists suggested face validity (mean 8.7 out of 10, SD 1.1). In terms of reliability, ACE-III demonstrated good internal consistency (Cronbach's alpha 0.786) and high inter-rater (ICC=0.936 [95%CI 0.921-0.941]) and test-retest reliability (ICC=0.972 (95%CI 0.958-0.981). The Greek version of ACE-III is a valid, and reliable screening tool that could be routinely employed perioperatively as a valid alternative to the MoCA test to distinguish the mild cognitively impaired from healthy elderly candidates for surgical interventions.

A novel scoring protocol reveals age-related differences in abstract compared to concrete thinking in cued autobiographical remembering

Human imagination has garnered growing interest in many fields. However, it remains unclear how to characterize different forms of imaginative thinking and how imagination differs between young and older adults. Here, we introduce a novel scoring protocol based on recent theoretical developments in the cognitive neuroscience of imagination to provide a broad tool with which to characterize imaginative thinking. The scoring protocol distinguishes between concrete/perceptual forms of imagination termed the “mind’s eye” and abstract/reflective forms of imagination termed the “mind’s mind.” The protocol also captures whether thoughts pertain to the self, others, or both. We applied this scoring protocol with high inter-rater reliability across two studies involving distinct participants and narrative-based imagination tasks. When compared to young adults, older adults showed a bias toward general content, which is a feature of the mind’s mind form of thinking while describing aloud their memories of specific, past events (Study 1). Further, older adults made fewer references to the self. In a separate study of only older adults (Study 2), increasing age was not associated with a bias toward the mind’s mind while describing specific past or future events. These results reveal that imaginative thinking can be characterized within the Mind’s Eye Mind’s Mind framework, with implications for understanding cognitively normal older age.

Medical Marijuana Documentation Practices in Patient Electronic Health Records: Retrospective Observational Study Using Smart Data Elements and a Review of Medical Records.

Medical marijuana (MMJ) is available in Pennsylvania, and participation in the state-regulated program requires patient registration and receiving certification by an approved physician. Currently, no integration of MMJ certification data with health records exists in Pennsylvania that would allow clinicians to rapidly identify patients using MMJ, as exists with other scheduled drugs. This absence of a formal data sharing structure necessitates tools aiding in consistent documentation practices to enable comprehensive patient care. Customized smart data elements (SDEs) were made available to clinicians at an integrated health system, Geisinger, following MMJ legalization in Pennsylvania. The purpose of this project was to examine and contextualize the use of MMJ SDEs in the Geisinger population. We accomplished this goal by developing a systematic protocol for review of medical records and creating a tool that resulted in consistent human data extraction. We developed a protocol for reviewing medical records for extracting MMJ-related information. The protocol was developed between August and December of 2022 and focused on a patient group that received one of several MMJ SDEs between January 25, 2019, and May 26, 2022. Characteristics were first identified on a pilot sample (n=5), which were then iteratively reviewed to optimize for consistency. Following the pilot, 2 reviewers were assigned 200 randomly selected patients' medical records, with a third reviewer examining a subsample (n=30) to determine reliability. We then summarized the clinician- and patient-level features from 156 medical records with a table-format SDE that best captured MMJ information. We found the review protocol for medical records was feasible for those with minimal medical background to complete, with high interrater reliability (κ=0.966; P<.001; odds ratio 0.97, 95% CI 0.954-0.978). MMJ certification was largely documented by nurses and medical assistants (n=138, 88.5%) and typically within primary care settings (n=107, 68.6%). The SDE has 6 preset field prompts with heterogeneous documentation completion rates, including certifying conditions (n=146, 93.6%), product (n=145, 92.9%), authorized dispensary (n=137, 87.8%), active ingredient (n=130, 83.3%), certifying provider (n=96, 61.5%), and dosage (n=48, 30.8%). We found preset fields were overall well-recorded (mean 76.6%, SD 23.7% across all fields). Primary diagnostic codes recorded at documentation encounters varied, with the most frequent being routine examinations and testing (n=34, 21.8%), musculoskeletal or nervous conditions, and signs and symptoms not classified elsewhere (n=21, 13.5%). This method of reviewing medical records yields high-quality data extraction that can serve as a model for other health record inquiries. Our evaluation showed relatively high completeness of SDE fields, primarily by clinical staff responsible for rooming patients, with an overview of conditions under which MMJ is documented. Improving the adoption and fidelity of SDE data collection may present a valuable data source for future research on patient MMJ use, treatment efficacy, and outcomes.

A visual scale to rate amygdalar atrophy on MRI.

Visual rating scales are routinely used in clinical radiology to assess brain atrophy on scans of patients with suspected neurodegenerative conditions. Limbic predominant age-related TDP-43 encephalopathy (LATE) has recently been described, featuring early and severe atrophy of the amygdala. However, there is currently no scoring system specifically designed to assess amygdalar atrophy on MRI. to develop and validate a visual rating scale for amygdalar atrophy. Stringent criteria were developed for no, mild/moderate, and severe amygdalar atrophy based on axial and coronal volumetric T1-weighted MRI scans. Inter- and intra-rater reliabilities were estimated by three independent expert neuroradiologists in 100 randomly selected scans from the Geneva Memory Center cohort selected to be representative of the variability of medial temporal atrophy. Convergent validity was evaluated versus amygdalar volumes extracted by FreeSurfer on 1943 consecutive patients. Criterion validity versus autopsy-confirmed LATE neuropathologic changes were studied in the pathological subset of the Alzheimer's Disease Neuroimaging Initiative (ADNI) cohort (N = 96). Intra- and inter-rater agreements of amygdalar visual ratings were between substantial and almost perfect (weighted Cohen's Kappa 0.71 to 0.93). Visual ratings were strongly associated with amygdalar volumes (p ≤ 0.001 on the Kruskal-Wallis test). LATE neuropathologic changes were associated with visual ratings of amygdalar atrophy (p = 0.057 on a test for trend). The proposed visual amygdalar atrophy scale is a reliable and valid tool to assess amygdalar atrophy on MRI and can be a useful adjunct in routine radiological reporting. Question Assessment of amygdalar atrophy is crucial for diagnosing neurodegenerative diseases, as the limbic predominant age-related TDP-43 encephalopathy, yet no validated visual rating scale exists. Findings The proposed amygdalar atrophy scale demonstrated high intra-rater and inter-rater reliability, strong correlation with amygdalar volumetry, and association with limbic predominant age-related TDP-43 encephalopathy (LATE). Clinical relevance The amygdalar atrophy scale provides a reliable practical assessment tool that enhances diagnostic accuracy for dementia-related conditions, particularly aiding in identifying limbic predominant age-related TDP-43 encephalopathy.

STRIAA (Severity TRichoscopy Index Alopecia Areata): Validation of a Novel Trichoscopic Tool for Evaluation of Alopecia Areata.

Alopecia areata (AA) is a non-scarring autoimmune disease characterized by patchy hair loss. The aim of this study was to validate a novel trichoscopic scoring tool, the Severity TRichoscopy Index for Alopecia Areata (STRIAA), for rapid assessment of AA severity. Anonymized images from 340 patients were scored by two independent raters who analyzed four scalp areas (vertex, occipital, and left and right parietal) for trichoscopic signs: black dots, yellow dots, exclamation mark hairs, broken hairs, and short vellus hairs. Scores (0-3) were assigned according to the number of trichoscopic signs per area, resulting in a total STRIAA score out of 60. STRIAA showed high interrater reliability (Cronbach's alpha 0.99) and significant correlation with the Severity of Alopecia Tool (SALT) score (p < 0.001). Yellow and black dots were significantly associated with the SALT score. The STRIAA provides a rapid and comprehensive assessment of AA severity, complementing current assessment tools in clinical practice.

Mobility scale for acute stroke patients (MSAS): construct validity and reliability of the Italian scale.

The Mobility Scale for Acute Stroke Patients (MSAS) was developed to discriminate between the lower levels of mobility in acute stroke patients in the first two weeks post-onset. The present study aims to develop and validate an Italian version of the MSAS. The English version of the MSAS was translated into Italian according to international guidelines. We recruited 58 patients admitted to a rehabilitation facility within 14 days of the stroke onset and administered the MSAS to test the internal consistency, concurrent validity, reliability, and responsiveness of the scale. As for internal consistency, Cronbach's alpha showed values equal to 0.96, and the alpha deleted analysis indicated that all the items have reason to exist. Concurrent validity showed statistically significant data for all sub-scales. The Italian version of the MSAS showed significant and high inter-rater reliabilities (all ICCs ≥ 0.75). The MSAS also showed excellent test-retest reliabilities (all ICCs ≥ 0.90). The reactivity of the scale, evaluated through the Student's t-test for paired samples on a sub-sample of 32 patients, showed statistically significant improvements over time for all items and the total scale. The results show that the scale is stable and reliable both in the evaluation after 24h and between different operators. High internal consistency and a strong concurrent validity were found. The scale also proved able to detect the improvement obtained by patients following the rehabilitation treatments.

Standard Setting for Assessments in Anatomy using the Angoff Method in a Medical School in Sri Lanka

Introduction: Anatomy is a fundamental subject in medical education. A reliable and valid assessment tool is crucial for evaluating students' understanding. The Angoff method, an absolute method of standard-setting has been widely used in various educational contexts. This study aimed to develop and implement a standard-setting process for an assessment in Anatomy using the Angoff method in a Sri Lankan medical school. Methods: Five Anatomy faculty members including two senior lecturers and three temporary assistant lecturers who were recent graduates participated in the standard-setting process. They were presented with each item on the anatomy assessment and asked to estimate the percentage of minimally competent students who would answer the item correctly. Their judgments were then used to determine the cut score for the assessment. Results: The Angoff method yielded a cut score of 56.06 with a cutoff value of 8.41 and a standard error of judgment of 0.18 (1.2%). There was a high interrater reliability (0.89). Conclusion: This study demonstrated the feasibility and effectiveness of using the Angoff method for standard setting in an anatomy assessment in a Sri Lankan medical school. The set cut score provides a benchmark for evaluating students' performance and ensuring that they possess the necessary anatomical knowledge to progress in their medical education.

The Italian version of the Postural Assessment Scale for Stroke Patients (PASS): transcultural translation and validation.

The Postural Assessment Scale for Stroke Patients (PASS) is commonly used by health professionals in Italy in several different translations. This study aimed to provide a validated version in Italian. The main focus is on the evaluator, to guarantee a uniform application and interpretation of the statements and scoring for each item in the Italian context. A standardized protocol was used for the translation and cross-cultural adaptation. A pilot study conducted using the first draft of the scale led to a revised version, PASS-IT. A principal component analysis (PCA) was performed. The correlation with the Trunk Control Test (TCT) was examined for concurrent validity. In addition, the relationship with the Barthel Index (BI) and the Functional Ambulation Categories (FAC) was tested. Patients with recent stroke were tested for intra-rater (N = 49) and inter-rater agreement (N = 30). Cronbach's alpha, item-to-total correlation, corrected inter-item correlation, the intraclass correlation coefficient (ICC), and measurement error were used to evaluate internal consistency and intra-/inter-rater reliability. The PCA showed a two-dimensional structure, with high reliability in both subsections ("non-weight-bearing" α = 0.865; "weight-bearing" α = 0.949). A strong correlation (ρ > 0.80) was found with the TCT, the BI, and the FAC. The PASS-IT showed high internal consistency, intra-rater (ICC = 0.942) and inter-rater reliability (ICC = 0.940). The PASS-IT is a recommended scale, suitable for clinical practice and research in the acute and subacute stage. The introduction of operating instructions resulted in the uniform application. A different order of the items allows faster administration, reducing changes of posture.

Localizing and Tailoring the Debriefing Assessment for Simulation in Healthcare to Optimize Fit.

Introduction Debriefing in healthcare simulation is helpful inreinforcing learning objectives, closing performance gaps, and improving future practice and patient care. The Debriefing Assessment for Simulation in Healthcare (DASH) is a validated tool. However, localized rater training for the DASH has not been described. We sought to augment DASH anchors with localized notations, localize DASH rater training, assess localized DASH correlation with other debriefing practices/factors, and assess reliability of the localized tool. Methods This study was conducted at SimTiki Simulation Center, John A. Burns School of Medicine, University of Hawai'i at Mānoa. Three simulation experts without DASH training developed a list of debriefing best practices/factors, reviewed the DASH handbook, and transcribed the DASH Rater Long Form version with example behaviors into a rating document. Research assistants recorded best practices/factor data from archived debriefing videos. Simulation experts independently scored debriefings, resolved discrepancies and added localized criteria to the DASH. Rater calibration was completed with an intraclass correlation (ICC) of 0.884. Raters then independently scored 43debriefings recorded during July-December 2022. DASH scores were compared to observed debriefing best practices/factors. Results The overall DASH behaviors ICC agreement was 0.810 and consistency was 0.825. Behavior scores ranged from 2.45 (SD 0.70) to 4.42 (SD 0.81). The three lowest scoring DASH behaviors were 2A (2.45), 4A (3.41), and 3B (3.44). In behavior 2B regarding realism concerns, there was significant inconsistency in the use of the not applicable (NA) designation. DASH and best practices/factors construct correlation supported convergent validity. Conclusion High interrater reliability followed localized rater training and the addition of localized notations to DASH. Correlation with debriefing practices/factors strengthens DASH validity evidence. Lower DASH behavior scores reported generally suggest that the interpretation of DASH scores is best contextualized in a shared localized DASH construct. A comparison of DASH numerical scores between institutions, with different raters, and different cultures may not reflect absolute debriefing quality.

Using Electronic Health Records to Identify the Daily Primary Provider During Hospitalization.

Providers vary in their impact on clinical outcomes, but this is rarely accounted for in healthcare research. By failing to identify the provider responsible for a patient's care, investigators miss an opportunity to account for nonrandom variation in outcomes. Prior methods of identifying responsible providers have relied on manual chart review, which is time-consuming and expensive, or analysis of claims data, which has been demonstrated to be inaccurate. To address these gaps, we sought to develop an algorithm using electronic health record (EHR) data to identify the responsible provider for each day of a patient's hospitalization. A multicenter retrospective cohort study. Midwest healthcare system. Hospitalized patients and their providers. None. We first confirmed high inter-rater reliability of manual chart review to identify the responsible provider. Using manual chart review as the gold standard, we then assessed the accuracy of an automated algorithm in a set of randomly selected patients. The agreement between two independent physicians in their determination of the responsible provider by chart review was 100%. Among 200 randomly selected patients, the algorithm identified the same responsible provider as the physician chart reviewer on 93% (3372/3626; 95% CI, 92-94%) of patient-days. Readily available EHR data can be used to assign patients to providers daily with a high degree of accuracy. This methodology could be applied in healthcare research to identify sources of variation other than the intervention being studied.

A qualitative study of attitudes and perceptions of smoking cessation medication among patients with cancer.

Among patients with cancer, tobacco use remains high. Nearly half who report smoking at diagnosis continue to smoke following diagnosis, and use of smoking cessation medication remains low. This studyexplores attitudes, barriers, and preferences for smoking cessation medication among patients enrolled in a tobacco treatment trial in the context of cancer care. This is a secondary qualitative analysis of the Smokefree Support Study. Of 221 participants who completed the 6-month follow-up survey, n = 72 participants were randomly selected for an exit interview. Interviews were conducted using a semi-structured interview guide, recorded, transcribed, and individually coded using NVivo 11 until high interrater reliability was reached (Kappa > 0.86). Participants (55.6% female) were predominantly White (83.3%) and on average 60years old. Many participants held negative beliefs about smoking cessation medications, primarily concerns about safety and side effects in the context of cancer treatment. Lack of understanding interactions with ongoing chemotherapy and access were the most common barriers reported by those who did not use cessation medication during the study. For those who did use medication during the study, positive outcomes included craving reduction, ease of use and access, and psychological benefits. The findings suggest a knowledge gap in the safety and effectiveness of smoking cessation medications, specifically in relation to cancer care. Addressing this knowledge gap may increase medication uptake, adherence, and quit rates. Cancer care providers are seen as instrumental in emphasizing the importance and safety of smoking cessation medications as well as patient access concerns. Tobacco use among patients with cancer remains high and has serious clinical implications. Tobacco cessation is associated with decreased treatment-related toxicity, decreased risk of second primary cancers, and increased survival rates. Smoking cessation medications are effective, yet patients with cancer have low usage rates. Elucidating reasons associated with smoking cessation medication nonadherence will assist cancer care providers in better addressing factors that increase adherence rates.

Simplifying the Animal Welfare Assessment Grid for enhanced accessibility.

Ensuring animal welfare is essential for both the well-being of zoo animals and the effective management of zoological facilities. This study introduces the Simplified Animal Welfare Assessment Grid (S-AWAG), a streamlined adaptation of the original AWAG framework that integrates the Five Domains Model with an observation-based approach. Designed for non-expert users, S-AWAG focuses on easily observable welfare indicators-such as health and environmental conditions-making it particularly suitable for small, private zoos, including petting zoos, roadside zoos, indoor zoos, and animal cafés. We conducted welfare assessments on 304 animals from 11 species across nine zoos in South Korea. The results revealed significant differences in welfare standards between accredited and non-accredited zoos, with accredited facilities consistently demonstrating better welfare conditions (p < 0.05). The tool exhibited high inter-rater reliability (IRR = 0.839), confirming its consistency across assessors with varying levels of expertise and ensuring reliable and accurate results. Pearson correlation analysis identified strong positive associations between health and environmental factors, reinforcing the comprehensive nature of the tool's evaluation approach. With its user-friendly, efficient, and adaptable design, S-AWAG has the potential to improve animal welfare standards not only in South Korea but also globally, particularly in smaller, resource-constrained facilities.

CIPAS Scale: Pilot Test for Evaluating Interaction Quality and Secure Attachment Potential

The adult's sensitivity to the child's communicative signals, their prompt and appropriate response to the child's needs for security, protection, and comfort, along with their ability to regulate emotions in the face of childhood stress, are parenting skills associated with attachment quality. Various instruments and observation scales code these constructs. Furthermore, developmental psychology has made significant contributions regarding the affective and communicative exchanges between parents and children during early stages of life and their implications for the formation of emotional bonds. The objective of this study is to describe the development and preliminary analysis of a new observation scale called: Quality of Interaction and Secure Attachment Potential (CIPAS), which aims to simply integrate the different parameters of the interactive relationship between attachment figures and children in early childhood. The construction process and initial psychometric analyses from a pilot sample are presented. The results showed very high inter-rater reliability, with an overall kappa value of κ=0.88. The relevance of CIPAS is discussed not only as a valuable tool for developmental psychology research but also as an effective instrument to guide educational practices in promoting secure attachment and quality interaction in young children.

Evaluating the reliability and validity of a Chinese version of the performance-oriented mobility assessment among patients with chronic stroke.

The Performance-Oriented Mobility Assessment (POMA) is a reliable instrument for evaluating the mobility (balance and gait) of patients with chronic stroke to manage their risk of falling; however, it has not been validated among Chinese patients with stroke. This study aimed to evaluate the reliability and validity of the Chinese POMA in patients with stroke. The POMA was applied to volunteer patients with stroke from the Shanghai MCC Hospital. The patients underwent the Chinese POMA, Berg balance scale (BBS), and timed up and go (TUG) tests on the first day of inpatient treatment. The same physician repeated the tests the next day to assess test-retest reliability, and upon the patient's discharge from the inpatient department, two different physicians measured inter-rater reliability. The study involved 76 patients with stroke (age: 62.04 ± 9.76 years; 34.2% female). The results showed that the Chinese POMA had good overall internal consistency (σ=. 875), with a moderate consistency between its two subscales (balance σ = 0.875; gait σ = 0.668). The individual items showed high test-retest (ICC = 0.997) and inter-rater reliability (ICC = 0.988). The content validity test showed high correlations between the Chinese POMA, the BBS (rs = 0.70), and the TUG (rs = -0.75). However, the confirmatory factor analysis suggested that the two-factor model (balance and gait) was mediocre. The Chinese POMA showed acceptable reliability and validity for evaluating mobility (balance and gait) in Chinese patients with stroke in terms of their risk of falling. However, further evaluation of the two-factor model (balance and gait) is required.

Achieving Inter-Rater Agreement and Inter-Rater Reliability to Assess Fidelity of an Occupation-Based Coaching (OBC) Clinical Trial Intervention

Introduction: Establishing inter-rater agreement and reliability ascertains that multiple raters consistently evaluate observed interventions to ensure that clinical research protocols are delivered as intended by the trial protocol. Purpose: Using the Guidelines for Reporting Reliability and Agreement Studies, we (a) exemplified the steps to establish inter-rater reliability and inter-rater agreement on the occupation-based coaching Video Evaluation Tool and (b) evaluated best practices that promoted high inter-rater reliability and inter-rater agreement between blinded raters prior to starting a pilot randomized controlled trial. The randomized controlled trial examined the preliminary effectiveness of occupation-based coaching via telehealth for rural families with children living with type 1 diabetes to improve family quality of life, participation, self-efficacy, and child health outcomes. Method: We created a library of 13 occupation-based coaching videos portraying a range of evaluations, scores, and ratings. The inter-rater agreement and reliability on the occupation-based coaching Video Evaluation Tool were established through the iterations of (a) blinded rater training, (b) data collection using the tool, and (c) statistical analysis using Cohen’s kappa and Cronbach’s alpha. Findings: Occurrence and Non-Occurrence Checklist (κ = 0.881, p &lt; 0.001); “Caregiver Talk” and “Interventionist Talk Analysis” (ICC = 0.991–0.999, p &lt; 0.001); Evidence of Independent Capacity Rating (ICC = 0.867 p = 0.006). Conclusion: Strong inter-rater reliability and inter-rater agreement was established by engaging two blinded raters through multifaceted training, integrating real-life clients and contexts into the instrumentation and training, and precisely defined rubric criteria. By employing such practices, high inter-rater reliability and agreement can be achieved in clinical research involving interventions and instruments that are highly subjective and individualized. To ascertain greater scientific confidence in the intervention effect, developing a multidomain fidelity framework and establishing high inter-rater agreement and reliability in the instruments a priori to implementation of clinical trials are necessary.