Inter-rater Reliability Analysis Research Articles

Online Administration of the Test of Narrative Language-Second Edition: Psychometrics and Considerations for Remote Assessment.

Our aim was to evaluate the psychometric properties of the online administered format of the Test of Narrative Language-Second Edition (TNL-2; Gillam & Pearson, 2017), given the importance of assessing children's narrative ability and considerable absence of psychometric studies of spoken language assessments administered online. The TNL-2 was administered to 357 school-age children at risk for language and literacy difficulties as part of a randomized controlled trial, across three annual cohorts, at three time points (pretest, posttest, and 5-month follow-up). Cohort 3 students were tested using an online format at posttest and at follow-up. We compared the Cronbach's alpha internal consistency reliability of the TNL-2 online testing scores with in-person scores from TNL-2 normative data and Cohort 3 in-person testing at pretest, and interrater reliability for Cohort 3 across test points. In addition, we examined measurement invariance across test occasions and the criterion validity of the TNL-2, the latter based on its correlations with narrative sample measures (Mean Length of Utterance in words and the Monitoring Indicators of Scholarly Language rubric). Internal consistency reliability, interrater reliability, and measurement invariance analyses of the online and in-person administration of the TNL-2 yielded similar outcomes. The criterion validity of the TNL-2 was found to be good. TNL-2 psychometric properties from online administration were generally in the good range and were not significantly different from in-person testing. When administered online using standardized procedures, the TNL-2 is valid and reliable for use in assessing narrative language proficiency in school-age children at risk for language and learning difficulties.

EXPLORATION OF ONLINE DISCUSSION FORUM ON READING EST TEXTS FOR ESL LEARNERS

Purpose: This study explored ESL learners’ utilization of Online Discussion Forums (ODFs) in Interactive Reading for Academic Disciplines (iREAD) to assist reading of online scientific texts within an English for Specific Purposes course at a public university. Method: This qualitative research employed two data collection methods during lessons of a 14-week semester. The first was analysis of ODF threads in iREAD from 30 ESL learners of various engineering programs at a public university in the East coast of Malaysia. Twelve from 30 ESL students were purposively selected to participate in focus group interviews. Interactive Analysis Model was used to analyse the threads, whilst the interviews transcripts were analysed thematically. NVivo Pro 12 software was used to manage the data and the generated themes were later analysed using inter-rater reliability analysis. The Kappa values of the themes indicated near perfect scores, to verify the reliability of the identified themes. Findings: The findings revealed the positive capability of ODF as a tool in facilitating learners’ reading comprehension of English for Science and Technical (EST) online materials. Analysis of the data on ODF threads confirm that students have the capabilities to engage in higher level of discussions and debate to aid their understanding. The themes that emerged from the interviews reflect positive impacts to facilitate reading of EST online materials, where learners were able to work collaboratively and increase knowledge and understand better. In addition, significance of EST materials was important in providing exposure to students’ background knowledge of their content courses. The notion of social constructivism and connectivism were reflected as learners constructed knowledge, negotiated, and synthesized ideas using the language across networked learning communities. Significance: An implication from this study suggests that exposure to EST online materials through communication and collaboration via ODF has strengthened students’ knowledge of the subject content. Additionally, learners’ comprehension of reading online EST materials has been enhanced through ODF because of the tool’s interactive heuristic approach to learning. This becomes significant for program developers and lecturers to combine both elements of ODF and material selection in lessons considering its positive impacts toward learning.

Team emergency assessment measure (TEAM) of non-technical skills: The Brazilian Portuguese version of the TEAM tool

ObjectivesThe aim of this study was to conduct the translation and cross-cultural adaptation of the original Team Emergency Assessment Measure (TEAM) tool into the Brazilian Portuguese language and investigate the internal consistency, inter-rater reliability, and concurrent validity of this new version (bp-TEAM). MethodsIndependent medical translators performed forward and backward translations of the TEAM tool between English and Portuguese, creating the bp-TEAM. The authors selected 23 videos from final-year medical students during in-situ emergency simulations. Three independent raters assessed all the videos using the bp-TEAM and provided a score for each of the 12 items of the tool. The authors assessed the internal consistency and the inter-rater reliability of the tool. ResultsRaters assessed all 23 videos. Internal consistency was assessed among the 11 items of the bp-TEAM from one rater, yielding a Cronbach's alpha of 0.89. inter-item correlation analysis yielded a mean correlation coefficient rho of 0.46. Inter-rater reliability analysis among the three raters yielded an intraclass correlation coefficient of 0.86 (95% CI 0.83‒0.89), p < 0.001. ConclusionThe Brazilian Portuguese version of the TEAM tool presented acceptable psychometric properties, similar to the original English version.

Frailty in community-dwelling older adults: reliability and validity of the Turkish version of the Gérontopôle frailty screening tool.

Recognizing frailty had a great importance in countries with an increasing geriatric population. The study aims to evaluate the reliability and validation of the Gérontopôle frailty screening tool (GFST), which was developed to screen frailty in the community-dwelling older adults, for the Turkish population. In this crosssectional study, participants who applied to the geriatrics outpatient clinic of a university hospital were included. Comprehensive geriatric assessments of all patients were performed. The validity of the GFST was tested by its concordance with the FRAIL scale. Test-retest and interrater reliability analyses were made. Ninety-six patients were included in the study. Sixty-one and five-tenths percent (n = 59) of them are females. The median age was 72.0 (IQR: 10.0). There was a moderate concordance between the GFST and the FRAIL scale (Cohen's kappa: 0.566, p < 0.001). The Gérontopôle frailty screening tool interrater and retest reliabilities were excellent (Cohen's kappa: 0.814, p < 0.001 and 1.0, p < 0.001; respectively). The sensitivity of the GFST determined according to the FRAIL scale is 69.39%, the specificity is 86.36%, the positive predictive value is 85.00%, and the negative predictive value is 71.70%. The Gérontopôle frailty screening tool, which can be used by all healthcare professionals, is a valid and reliable tool for the Turkish geriatric population.

Model to Cope With Requirements Engineering Issues for Software Development Outsourcing

The anticipated benefits of Software Development Outsourcing (SDO) are not achieved in case of several projects because of the issues that emanate from Requirements Engineering (RE) process. This research work presents a Requirements Engineering Practices (REP) model to cope with the customarily occurring issues of the RE process for SDO. To formulate the model, five workshops have been conducted and Root Cause Analysis has been performed by considering 43 commonly occurring SDO RE process issues, and 147 RE practices to tackle the issues. To discover the root causes for commonly transpiring issues, 5- Whys technique has been employed. The relevant RE practices that can be used to deal with the root causes, have been endorsed by applying Brainstorming technique. For the 43 frequently occurring issues, 89 root causes have been discovered. Afterwards, 124 relevant RE practices have been recommended to eradicate the root causes and hence to address the corresponding issues. Thus, REP model postulates the root causes for commonly occurring issues of the SDO RE process, maps the root causes to the relevant best RE practices to address the corresponding issues. The model has been evaluated by an expert panel and evaluation results have been analysed through Inter-Rater Reliability analysis and Analysis of Means. The REP model supports the RE process for SDO by i). evading the adoption of random and inappropriate RE practices for dealing with the common issues of the process, ii) helping to attain the expected benefits of SDO.

Review, Assess, Classify, and Evaluate (RACE): a framework for studying m-health apps and its application for opioid apps.

The proliferation of m-health interventions has led to a growing research area of app analysis. We derived RACE (Review, Assess, Classify, and Evaluate) framework through the integration of existing methodologies for the purpose of analyzing m-health apps, and applied it to study opioid apps. The 3-step RACE framework integrates established methods and evidence-based criteria used in a successive manner to identify and analyze m-health apps: the Preferred Reporting Items for Systematic Reviews and Meta-Analyses, inter-rater reliability analysis, and Nickerson-Varshney-Muntermann taxonomy. Using RACE, 153 opioid apps were identified, assessed, and classified leading to dimensions of Target Audience, Key Function, Operation, Security & Privacy, and Impact, with Cohen's kappa < 1.0 suggesting subjectivity in app narrative assessments. The most common functions were education (24%), prescription (16%), reminder-monitoring-support (13%), and treatment & recovery (37%). A majority are passive apps (56%). The target audience are patients (49%), healthcare professionals (39%), and others (12%). Security & Privacy is evident in 84% apps. Applying the 3-step RACE framework revealed patterns and gaps in opioid apps leading to systematization of knowledge. Lessons learned can be applied to the study of m-health apps for other health conditions. With over 350 000 existing and emerging m-health apps, RACE shows promise as a robust and replicable framework for analyzing m-health apps for specific health conditions. Future research can utilize the RACE framework toward understanding the dimensions and characteristics of existing m-health apps to inform best practices for collaborative, connected and continued care.

Playing Edcraft at Home: Gamified Online Learning for Recycling Intention during Lockdown

Background: Gamification is an innovative approach to engaging in activities that people believe as less interesting. Recycling has been an issue not taken aware by the people in environmental sustainability. There are substantial studies on recycling intention due to the continual growth of unethical and unsustainable waste disposal. Creative approaches to recycling awareness activities should be made to fulfil youths’ increasing interest in and demand for recycling. The main objective of this study is to explore the factors related to youths’ recycling intentions after experiencing a gamified online recycling learning activity, Edcraft Gamified Learning (EGL). Gamified recycling education is believed to be a practical and engaging approach for youths. Methods: 100 students participated in EGL, consisting of two levels of plastic crafting and recycling activities. They experienced online EGL at home between May and September in 2020, during the COVID-19 pandemic total lockdown in Malaysia, namely, Movement Control Order (MCO). 29 participants were selected to participate in five focus group discussions (FGDs) with five to eight participants per session to explore their opinions towards gamified learning, motivation and recycling intention. Results: This paper reports the findings of the FGDs. A codebook was developed based on the codes from the FGD feedback. The codes were rated by two raters, followed by an assessment of inter-rater reliability and thematic analysis. The findings emerged and were confirmed with four themes as factors that influence recycling intention. They are gameful experience, social influence, intrinsic motivation, and extrinsic motivation. Conclusion: The dependent variable, recycling intention, was connected to the four themes to verify the conceptual framework. One limitation of the study was the design of the EGL activity, which was only carried out over two days with two levels of gamified recycling education, as students had concurrent academic online classes during that period.

Translating and Assessing the Reliability of the Persian Version of the Profile of Occupational Engagement

Background and Objectives: The profile of occupational engagement in People with Severe Mental Illness (POES) is one of the time-use-based instruments, which can uncover the rhythm of activity and rest, as well as the variety and range of meaningful occupations. The current study aimed to report the translation process and assess the inter-rater reliability of the Persian version of POES. Methods: In the present study, the translation process took place in three phases: forward-translation, backward-translation, and two expert panels review. To assess inter-rater reliability, all the POES forms were rated by the first and the third author. Results: The data achieved from 47 patients with schizophrenia were carefully analyzed. The inter-rater reliability analysis shows the intra-class correlation values were 0.86 in single and 0.92 in average measures. Conclusion: The reliability results are entirely consistent with the previous studies on POES.

Playing Edcraft at Home: Gamified Online Learning for Recycling Intention during Lockdown

Background: Gamification is an innovative approach to engaging in activities that people believe as less interesting. Recycling has been an issue not taken aware by the people in environmental sustainability. There are substantial studies on recycling intention due to the continual growth of unethical and unsustainable waste disposal. Creative approaches to recycling awareness activities should be made to fulfil youths’ increasing interest in and demand for recycling. The main objective of this study is to explore the factors related to youths’ recycling intentions after experiencing a gamified online recycling learning activity, Edcraft Gamified Learning (EGL). Gamified recycling education is believed to be a practical and engaging approach for youths. Methods: 100 students participated in EGL, consisting of two levels of plastic crafting and recycling activities. They experienced online EGL at home between May and September in 2020, during the COVID-19 pandemic total lockdown in Malaysia, namely, Movement Control Order (MCO). 29 participants were selected to participate in five focus group discussions (FGDs) with five to eight participants per session to explore their opinions towards gamified learning, motivation and recycling intention. Results: This paper reports the findings of the FGDs. A codebook was developed based on the codes from the FGD feedback. The codes were rated by two raters, followed by an assessment of inter-rater reliability and thematic analysis. The findings emerged and were confirmed with four themes as factors that influence recycling intention. They are gameful experience, social influence, intrinsic motivation, and extrinsic motivation. Conclusion: The dependent variable, recycling intention, was connected to the four themes to verify the conceptual framework. One limitation of the study was the design of the EGL activity, which was only carried out over two days with two levels of gamified recycling education, as students had concurrent academic online classes during that period.

Comparative Analysis of Lumbar Spine Vertebral Morphology Between Māori and New Zealand Europeans: A Computed Tomography Study.

Most existing studies of lumbar anatomy do not consider ethnic influence and recruit mostly white participants. Recent studies have considered other populations; however, none have assessed Māori, the indigenous people of New Zealand (NZ). A computed tomography study of vertebral body (VB) and canal dimensions was performed for lumbar vertebrae of Māori and NZ European patients to evaluate for ethnic variation. Lumbar vertebrae from 196 patients were measured using computed tomography. After interrater and intrarater reliability analyses, a single trained examiner measured VB heights, VB lengths, segmental angle, pedicle height and width, and vertebral canal length (VCL) and vertebral canal width for each level. Canal:body ratio was calculated. Demographic data recorded included age, sex, and ethnicity. VCL remained relatively constant through the lumbar spine; canal width increased to a maximum of 28.2 mm at L5. Canal:body ratios and pedicle height decreased while pedicle width increased to a maximum of 16.1 mm at L5. There were few differences between Māori and NZ Europeans except at the L5 level, where VCL and canal:body ratio were larger in NZ Europeans (P < 0.05), and pedicle height, width, and VB pediculolaminar length were larger in Māori (P < 0.05). Females had generally smaller measurements and age was a positive predictor of measured values (P < 0.05). This study is the first to characterize lumbar anatomy in a Māori cohort. Adequately powered results demonstrated few differences between Māori and NZ Europeans. Isolated differences observed at L5 may be due to sacropelvic differences, which represent an area for further investigation. 3. Diagnosis, surgical planning, ethnic differences.

Are English-language online patient education materials related to breast cancer risk assessment understandable, readable, and actionable?

PurposeTo evaluate the readability, understandability, and actionability of online patient education materials (OPEM) related to breast cancer risk assessment. Material and methodsWe queried seven English-language search terms related to breast cancer risk assessment: breast cancer high-risk, breast cancer risk factors, breast cancer family history, BRCA, breast cancer risk assessment, Tyrer-Cuzick, and Gail model. Websites were categorized as: academic/hospital-based, commercial, government, non-profit or academic based on the organization hosting the site. Grade-level readability of qualifying websites and categories was determined using readability metrics and generalized estimating equations based on written content only. Readability scores were compared to the recommended parameters set by the American Medical Association (AMA). Understandability and actionability of OPEM related to breast cancer high-risk were evaluated using the Patient Education Materials Assessment Tool (PEMAT) and compared to criteria set at ≥70%. Descriptive statistics and inter-rater reliability analysis were utilized. Results343 websites were identified, of which 162 met study inclusion criteria. The average grade readability score was 12.1 across all websites (range 10.8–13.4). No website met the AMA recommendation. Commercial websites demonstrated the highest overall average readability of 13.1. Of the 26 websites related to the search term breast cancer high-risk, the average understandability and actionability scores were 62% and 34% respectively, both below criteria. ConclusionsOPEM on breast cancer risk assessment available to the general public do not meet criteria for readability, understandability, or actionability. To ensure patient comprehension of medical information online, future information should be published in simpler, more appropriate terms.

Increased anterior pelvic tilt in patients with acetabular retroversion compared to the general population: A radiographic and prevalence study

IntroductionThe prevalence of acetabular retroversion is sparsely investigated. It may be associated with increased anterior pelvic tilt. The purpose of this study was to investigate whether patients with symptomatic and radiographically verified acetabular retroversion demonstrated increased anterior pelvic tilt compared to a control group, and furthermore to evaluate the prevalence of acetabular retroversion in the general population. MethodsAnteroposterior pelvic radiographs in standing position of 111 patients with acetabular retroversion prior to anteverting periacetabular osteotomy (PAO) and 132 matched controls from the general Danish population were assessed. Pelvic tilt was assessed by the sacrococcygeal joint–symphysis distance and pelvic-tilt-ratio. Acetabular retroversion was defined as positive cross-over sign and posterior wall sign. Prior to assessments, interrater reliability analysis was performed. Measurements were agreed by two independent assessors. A nonparametric regression model was used to test between-group differences in median pelvic tilt. The prevalence was calculated as the ratio of subjects and hips with acetabular retroversion, respectively. ResultsThe patient group had significantly larger median anterior pelvic tilt of 14.3 mm in sacrococcygeal joint–symphysis distance and −0.08 in pelvic-tilt-ratio, compared to controls. The prevalence of subjects in the general population with either unilateral or bilateral acetabular retroversion was 24% and 18% for all hips. ConclusionOur data demonstrated that patients with symptomatic acetabular retroversion have increased anterior pelvic tilt compared to the general population. Radiographic sign of acetabular retroversion was highly prevalent in the general population. Implication for practiceIncreased anterior pelvic tilt should be considered when diagnosing and treating patients with hip pain, as symptoms may be related to the functional position of the pelvis and not necessarily solely come from the radiographic verified acetabular retroversion.

Interpretation of Thoracic Radiography Shows Large Discrepancies Depending on the Qualification of the Physician-Quantitative Evaluation of Interobserver Agreement in a Representative Emergency Department Scenario.

(1) Background: Chest radiography (CXR) is still a key diagnostic component in the emergency department (ED). Correct interpretation is essential since some pathologies require urgent treatment. This study quantifies potential discrepancies in CXR analysis between radiologists and non-radiology physicians in training with ED experience. (2) Methods: Nine differently qualified physicians (three board-certified radiologists [BCR], three radiology residents [RR], and three non-radiology residents involved in ED [NRR]) evaluated a series of 563 posterior-anterior CXR images by quantifying suspicion for four relevant pathologies: pleural effusion, pneumothorax, pneumonia, and pulmonary nodules. Reading results were noted separately for each hemithorax on a Likert scale (0–4; 0: no suspicion of pathology, 4: safe existence of pathology) adding up to a total of 40,536 reported pathology suspicions. Interrater reliability/correlation and Kruskal–Wallis tests were performed for statistical analysis. (3) Results: While interrater reliability was good among radiologists, major discrepancies between radiologists’ and non-radiologists’ reading results could be observed in all pathologies. Highest overall interrater agreement was found for pneumothorax detection and lowest agreement in raising suspicion for malignancy suspicious nodules. Pleural effusion and pneumonia were often suspected with indifferent choices (1–3). In terms of pneumothorax detection, all readers mainly decided for a clear option (0 or 4). Interrater reliability was usually higher when evaluating the right hemithorax (all pathologies except pneumothorax). (4) Conclusions: Quantified CXR interrater reliability analysis displays a general uncertainty and strongly depends on medical training. NRR can benefit from radiology reporting in terms of time efficiency and diagnostic accuracy. CXR evaluation of long-time trained ED specialists has not been tested.

Analysis of interrater reliability in age assessment of minors: how does expertise influence the evaluation?

Nowadays, the clinical forensic medical management of migration flows comprises the age assessment of unaccompanied minors. The process of age estimation is a fundamental pillar for legally ensuring the minors’ rights and their protection needs. The procedure is complex and involves different phases and actors, from medical doctors to law enforcement officers. The present study aimed to investigate the performance of Greulich and Pyle, Demirjian, and Mincer methods when performed by raters both trained and without training. Also, the interrater reliability within groups of raters from different areas of expertise was evaluated. A total of 36 participants were enrolled for this study, divided in two groups according to their level of experience with age estimation methods. Each rater was asked to allocate stages and standards for age assessment, evaluating ten orthopantomograms and ten hand-wrist roentgenograms. The interrater reliability expressed through the Fleiss Kappa coefficient and the agreement with the reference standard were calculated. The results showed that none of the categories analyzed could reach a good interrater reliability (ĸ > 0.8) for both methods. The study results highlighted variation and disagreement in the interpretation of the sample among raters and in the subsequent stages and standards allocation. In conclusion, the results of this study highlight that expertise does influence the reliability of the most utilized methods of age estimation of living individuals and stress the importance of proper training and practice, which could greatly increase the accuracy of age assessments.

Development and Validation of the Indonesian Version of Quality Nursing Work Environment (QNWE-I) Instrument

Background: Nursing shortages are a significant problem experienced by medical institutions worldwide as well as Indonesia. Understanding nurses’ work environment is a proper strategy for enhancing the retention and overcoming shortages of nurses. The Quality Nursing Work Environment (QNWE) is a well-known tool that has been commonly used to evaluate the working atmosphere of nurses. However, the availability and validated version of the instrument for hospital nurses in Indonesia is not confirmed.Purpose: This study aims to confirm the psychometric evaluation of the Indonesian version of QNWE instruments in hospital settings.Methods: A cross-sectional study design and psychometric analyses were administered. The 65-item, eight-domain questionnaire was translated, adapted, and face-validated using content validity index (CVI) analysis. The construct validity and internal consistency of the translated version were tested using confirmatory factor analysis (CFA) and Cronbach's Alpha to 334 hospital nurses. The intraclass correlation coefficient (ICC) was employed to determine test-retest and interrater reliability analysis to 33 nurses and 10 ratters in the pilot testing.Results: The CVI of the QNWE-I scale ranged from 0.86 to 1.00. The CFA determined an adequate model fit of the instrument. As to overall reliability, test-retest reliability of 0.90, ICC analysis of 0.82, and Cronbach's alpha of 0.96; was confirmed.Conclusions: The psychometric evaluation of the QNWE-I showed excellent acceptability, validity, and reliability. Therefore, the QNWE-I can serve as a comprehensive instrument for assessing the quality of nurses' work environments in Indonesia.

P14.17 Conventional MRI criteria differentiate true tumour progression from treatment-induced effects in irradiated WHO grade 3 and 4 gliomas

Abstract BACKGROUND Post-treatment radiological deterioration of patients with an irradiated high-grade (WHO grade 3 and 4) glioma (HGG) may be the result of true progressive disease (PD) or treatment-induced effects (TIE). Differentiation between these two entities is of great importance, but remains a diagnostic challenge. This study assesses the diagnostic value of conventional MRI characteristics to differentiate PD from TIE in treated HGGs. MATERIAL AND METHODS In this single-centre, retrospective cohort study, we included adult patients with a HGG, who were treated with radiotherapy and subsequently developed a new or increasing contrast-enhancing lesion on conventional follow-up MRI. TIE and PD were defined radiologically as stable/decreased for a minimum of six weeks or progressive according to the RANO criteria, and histologically as predominantly TIE without viable tumour or PD. Demographic and clinical data were retrieved. Twenty-one preselected MRI characteristics of the progressive lesions were assessed by two neuroradiologists. The statistical analysis included logistic regression to develop a) a full multivariable model b) a diagnostic model with model reduction, and a Cohen’s Kappa interrater reliability coefficient. RESULTS 210 patients (median age 61, IQR=54–68, 189 males) with 284 lesions were included, of which 141 (50%) had PD. Median time to PD was 2 (0.7–6.1) and to TIE 0.9 (0.7–3.5) months after RT. In multivariable modelling and after model reduction, the following determinants were significant diagnostic factors: Radiation dose (Odds ratio (OR)=0.68, p=0.017), longer time since radiotherapy (OR=3.56, p&amp;lt;0.0005), certain enhancement patterns (soap bubble enhancement: OR=2.63, p=0.003), isointense apparent diffusion coefficient-signal (OR=2.11, p=0.036), development of multiple new lesions (OR=1.68, p=0.088) and increased marginal enhancement (OR=2.04, p=0.027). ORs of &amp;gt;1 indicate higher odds of PD. The Hosmer &amp; Lemeshow test showed a good calibration (p=0.947) and the area under the ROC-curve was 0.722 (95%-CI=0.66–0.78). Interrater reliability analysis between neuroradiologists revealed moderate to near-perfect agreement for the significantly predictive items, but poor agreement for others. CONCLUSION In patients with irradiated high-grade gliomas, several characteristics from conventional MRI are significant predictors for the discrimination between true progression and treatment-induced effects. Interrater reliability for these characteristics was variable. Conventional MRI characteristics from this study should be incorporated into a multimodal diagnostic model that includes advanced imaging techniques. FUNDING INFORMATION Foundation Vrienden UMC Utrecht and The StophersenkankerNU Foundation.

How is Video Efficient to Diagnose Sport-Related Concussion? A Cross-Sectional Study in the French TOP14 Rugby Championship.

To describe the frequency of clinical signs and the accuracy of video diagnosis of sport-related concussion. An observational cross-sectional study. Videos from a database of all suspected sport-related concussion in TOP14 matches (French professional male rugby Championship) were used, from 2012 to 2015 seasons. The videos were analyzed by 4 observers, blinded to the concussion diagnosis, after a training phase, and an inter-rater reliability analysis. N/A. Video analysis for clinical signs and presumed diagnosis of sport-related concussion. The observer's diagnosis was compared with the team physicians' diagnosis. Four hundred seventeen videos were analyzed with 142 concussed players. Sport-related concussions happened mostly from an impact to the head (seen by the observers in 98.4%), against an opponent (73.2%), and during a tackle (74.6%). Video signs were observable in 98.6% for suspected loss of consciousness, tonic posturing (95%), ataxia (77.2%), dazed look (35.7%), abnormal behavior (18.2%), and seizures (96.2%). Ataxia was seen in 77.7% of concussed players, suspected loss of consciousness in 61.4%, dazed look in 63.2%, abnormal behavior in 55%, and tonic posturing in 7.1%. The observers diagnosed 79.8% of concussions. We described the frequency of video clinical signs of sport-related concussion, with a good accuracy of the blinded observers for the diagnosis. This emphasizes the importance of pitch-side video analysis as an extra tool for sport-related concussion diagnosis.

Playing Edcraft at Home: Gamified Online Learning for Recycling Intention during Lockdown.

Background: Recycling is a way to sustain ecosystems. There are substantial studies on recycling intention due to the continual growth of unethical and unsustainable waste disposal. Creative approaches to recycling awareness activities should be made to fulfil youths' increasing interest in and demand for recycling. The main objective of this study is to explore the factors related to youths' recycling intentions after experiencing a gamified online recycling learning activity, Edcraft Gamified Learning (EGL). Gamified recycling education is believed to be a practical and engaging approach for youths. Methods: 100 students participated in EGL, consisting of two levels of plastic crafting and recycling activities. They experienced online EGL at home between May and September in 2020, during the COVID-19 pandemic total lockdown in Malaysia, namely, Movement Control Order (MCO). 29 participants were selected to participate in five focus group discussions (FGDs) with five to eight participants per session to explore their opinions towards gamified learning, motivation and recycling intention. Results: This paper reports the findings of the FGDs. A codebook was developed based on the codes from the FGD feedback. The codes were rated by two raters, followed by an assessment of inter-rater reliability and thematic analysis. The findings emerged and were confirmed with four themes as factors that influence recycling intention. They are gameful experience, social influence, intrinsic motivation, and extrinsic motivation. Conclusion: The dependent variable, recycling intention, was connected to the four themes to verify the conceptual framework. One limitation of the study was the design of the EGL activity, which was only carried out over two days with two levels of gamified recycling education, as students had concurrent academic online classes during that period.

Inter-rater reliability of Böhler and Gissane angles in different calcaneal fracture according to the Essex-Lopresti and Sanders classifications

Objective: To evaluate the inter-rater reliability and intra-class correlation coefficients (ICC) of Böhler’s angle and the critical angle of Gissane in calcaneal fractures, stratified by severity and by the Essex-Lopresti and Sanders classifications. Methods: Retrospective study of radiographs obtained from 97 patients: 67 with calcaneal fractures and 30 with normal lateral radiographs (used as a control group). Böhler’s angle and the angle of Gissane were measured by six raters: two orthopedic surgery residents, two musculoskeletal radiologists, a foot and ankle surgery fellow, and a senior consultant in foot surgery. Statistical analysis of inter-rater reliability was performed for the two angles, in the sample overall and stratified by the different radiographic and CT subtypes of calcaneal fractures. Results: For the angle of Gissane, the ICC was at best 0.400 (95% CI: 0.250 to 0.581) for normal radiographs, with poor agreement across all classifications and severity stratifications. For Böhler’s angle, the ICC values indicated weak to moderate agreement, with the best reproducibility obtained for the overall sample (0.740; 95% CI: 0.673 to 0.801). In Sanders type 1 fractures, the ICC was 0.704 (95% CI: 0.397 to 0.940), and in Sanders type 2 fractures, 0.762 (95% CI: 0.634 to 0.870). Conclusion: Böhler’s angle is more reproducible than the critical angle of Gissane, with greater inter-rater reliability among fractures deemed less severe on the Sanders classification, although the overall ICC ranged from weak to moderate at best. Level of Evidence III; Case Control Study; Diagnostic Studies.

A tool to assess nontechnical skills of perfusionists in the cardiac operating room

ObjectivesThis study aimed to develop the Perfusionists' Intraoperative Non-Technical Skills tool, specifically to the perfusionists' context, and test its inter-rater reliability. MethodsAn expert panel was convened to review existing surgical nontechnical skills taxonomies and develop the Perfusionists' Intraoperative Non-Technical Skills tool. During a workshop held at a national meeting, perfusionists completed the Perfusionists’ Intraoperative Non-Technical Skills ratings after watching 4 videos displaying simulated cardiac operations. Two videos showed “good performance,” and 2 videos showed “poor performance.” Inter-rater reliability analysis was performed and intraclass correlation coefficient was reported. ResultsThe final version of the Perfusionists' Intraoperative Non-Technical Skills taxonomy contains 4 behavioral categories (decision making, situation awareness, task management and leadership, teamwork and communication) with 4 behavioral elements each. Categories and elements are rated using an 8-point Likert scale ranging from 0.5 to 4.0. A total of 60 perfusionist raters were included and the comparison between rating distribution on “poor performance” and “good performance” videos yielded a statistically significant difference between groups, with a P value less than .001. A similar difference was found in all behavioral categories and elements. Reliability analysis showed moderate inter-rater reliability across overall ratings (intraclass correlation coefficient, 0.735; 95% confidence interval, 0.674-0.796; P < .001). Similar inter-rater reliability was found when raters were stratified by experience level. ConclusionsThe Perfusionists' Intraoperative Non-Technical Skills tool presented moderate inter-rater reliability among perfusionists with varied levels of experience. This tool can be used to train and assess perfusionists in relevant nontechnical skills, with the potential to enhance safety and improve surgical outcomes.