Low Inter-rater Reliability Research Articles

What faculty write versus what students see? Perspectives on multiple-choice questions using Bloom’s taxonomy

Background Using revised Bloom’s taxonomy, some medical educators assume they can write multiple choice questions (MCQs) that specifically assess higher (analyze, apply) versus lower-order (recall) learning. The purpose of this study was to determine whether three key stakeholder groups (students, faculty, and education assessment experts) assign MCQs the same higher- or lower-order level. Methods In Phase 1, stakeholders’ groups assigned 90 MCQs to Bloom’s levels. In Phase 2, faculty wrote 25 MCQs specifically intended as higher- or lower-order. Then, 10 students assigned these questions to Bloom’s levels. Results In Phase 1, there was low interrater reliability within the student group (Krippendorf’s alpha = 0.37), the faculty group (alpha = 0.37), and among three groups (alpha = 0.34) when assigning questions as higher- or lower-order. The assessment team alone had high interrater reliability (alpha = 0.90). In Phase 2, 63% of students agreed with the faculty as to whether the MCQs were higher- or lower-order. There was low agreement between paired faculty and student ratings (Cohen’s Kappa range .098–.448, mean .256). Discussion For many questions, faculty and students did not agree whether the questions were lower- or higher-order. While faculty may try to target specific levels of knowledge or clinical reasoning, students may approach the questions differently than intended.

Translational Application of a Neuro-Scientific Multi-Modal Approach Into Forensic Psychiatric Evaluation: Why and How?

A prominent body of literature indicates that insanity evaluations, which are intended to provide influential expert reports for judges to reach a decision “beyond any reasonable doubt,” suffer from a low inter-rater reliability. This paper reviews the limitations of the classical approach to insanity evaluation and the criticisms to the introduction of neuro-scientific approach in court. Here, we explain why in our opinion these criticisms, that seriously hamper the translational implementation of neuroscience into the forensic setting, do not survive scientific scrutiny. Moreover, we discuss how the neuro-scientific multimodal approach may improve the inter-rater reliability in insanity evaluation. Critically, neuroscience does not aim to introduce a brain-based concept of insanity. Indeed, criteria for responsibility and insanity are and should remain clinical. Rather, following the falsificationist approach and the convergence of evidence principle, the neuro-scientific multimodal approach is being proposed as a way to improve reliability of insanity evaluation and to mitigate the influence of cognitive biases on the formulation of insanity opinions, with the final aim to reduce errors and controversies.

Neuropathological consensus criteria for the evaluation of Lewy pathology in post-mortem brains: a multi-centre study

Currently, the neuropathological diagnosis of Lewy body disease (LBD) may be stated according to several staging systems, which include the Braak Lewy body stages (Braak), the consensus criteria by McKeith and colleagues (McKeith), the modified McKeith system by Leverenz and colleagues (Leverenz), and the Unified Staging System by Beach and colleagues (Beach). All of these systems use semi-quantitative scoring (4- or 5-tier scales) of Lewy pathology (LP; i.e., Lewy bodies and Lewy neurites) in defined cortical and subcortical areas. While these systems are widely used, some suffer from low inter-rater reliability and/or an inability to unequivocally classify all cases with LP. To address these limitations, we devised a new system, the LP consensus criteria (LPC), which is based on the McKeith system, but applies a dichotomous approach for the scoring of LP (i.e., “absent” vs. “present”) and includes amygdala-predominant and olfactory-only stages. α-Synuclein-stained slides from brainstem, limbic system, neocortex, and olfactory bulb from a total of 34 cases with LP provided by the Newcastle Brain Tissue Resource (NBTR) and the University of Pennsylvania brain bank (UPBB) were scanned and assessed by 16 raters, who provided diagnostic categories for each case according to Braak, McKeith, Leverenz, Beach, and LPC systems. In addition, using LP scores available from neuropathological reports of LP cases from UPBB (n = 202) and NBTR (n = 134), JT (UPBB) and JA (NBTR) assigned categories according to all staging systems to these cases. McKeith, Leverenz, and LPC systems reached good (Krippendorff’s α ≈ 0.6), while both Braak and Beach systems had lower (Krippendorff’s α ≈ 0.4) inter-rater reliability, respectively. Using the LPC system, all cases could be unequivocally classified by the majority of raters, which was also seen for 97.1% when the Beach system was used. However, a considerable proportion of cases could not be classified when using Leverenz (11.8%), McKeith (26.5%), or Braak (29.4%) systems. The category of neocortical LP according to the LPC system was associated with a 5.9 OR (p < 0.0001) of dementia in the 134 NBTR cases and a 3.14 OR (p = 0.0001) in the 202 UPBB cases. We established that the LPC system has good reproducibility and allows classification of all cases into distinct categories. We expect that it will be reliable and useful in routine diagnostic practice and, therefore, suggest that it should be the standard future approach for the basic post-mortem evaluation of LP.

Quantification of Motor Function Post-Stroke Using Novel Combination of Wearable Inertial and Mechanomyographic Sensors.

Subjective clinical rating scales represent the gold-standard for diagnosis of motor function following stroke. In practice however, they suffer from well-recognized limitations including assessor variance, low inter-rater reliability and low resolution. Automated systems have been proposed for empirical quantification but have not significantly impacted clinical practice. We address translational challenges in this arena through: (1) implementation of a novel sensor suite combining inertial measurement and mechanomyography (MMG) to quantify hand and wrist motor function; and (2) introduction of a new range of signal features extracted from the suite to supplement predicted clinical scores. The wearable sensors, signal features, and machine learning algorithms have been combined to produce classified ratings from the Fugl-Meyer clinical assessment rating scale. Furthermore, we have designed the system to augment clinical rating with several sensor-derived supplementary features encompassing critical aspects of motor dysfunction (e.g. joint angle, muscle activity, etc.). Performance is validated through a large-scale study on a post-stroke cohort of 64 patients. Fugl-Meyer Assessment tasks were classified with 75% accuracy for gross motor tasks and 62% for hand/wrist motor tasks. Of greater import, supplementary features demonstrated concurrent validity with Fugl-Meyer ratings, evidencing their utility as new measures of motor function suited to automated assessment. Finally, the supplementary features also provide continuous measures of sub-components of motor function, offering the potential to complement low accuracy but well-validated clinical rating scales when high-quality motor outcome measures are required. We believe this work provides a basis for widespread clinical adoption of inertial-MMG sensor use for post-stroke clinical motor assessment.

Inter-Rater Variability of Reflux Finding Score Amongst Otolaryngologists

Laryngopharyngeal reflux (LPR) is an extra-oesophageal variant of gastro-oesophgeal reflux disease. Patients often do not present with the classic reflux symptoms of heartburn or regurgitation. Accurate diagnosis of LPR can be challenging. The reflux finding score (RFS) is system based on the assessment of eight parameters seen on fibre optic laryngoscopy, used to determine the presence and the severity of laryngopharyngeal reflux (LPR). Scoring the RFS is subjective and highly dependent on the examiner's eye and experience. In this study, we investigated the inter-rater reliability between three otolaryngologists scoring a large library of video-recorded laryngoscopies for RFS. To evaluate the usefulness of RFS in daily clinical practice by assessing inter-rater reliability among otolaryngologists when interpreting a bank of identical fibre optic laryngoscopy examinations. Three board-certified otolaryngologists with different subspecialist interests examined video-recorded fibre optic laryngoscopies of 193 patients with or without LPR symptoms and rated each video for RFS. Statistical analysis was performed. Results were compared to determine the inter-rater reliability. Fair to poor correlation was found between the three expert raters for total RFS score, as well as for RFS component items with nonbinary outcomes. For the dichotomous items, the inter-rater reliability was slight to moderate. Inter-rater correlation for determining whether an examination is pathological or nonpathological was fair. The RFS alone was not reliable for confirming the diagnosis of LPR, due to low inter-rater reliability and the subjective nature of the scoring system.

Virtual audits of the urban streetscape: comparing the inter-rater reliability of GigaPan\xae to Google Street View

BackgroundAlthough previous research has highlighted the association between the built environment and individual health, methodological challenges in assessing the built environment remain. In particular, many researchers have demonstrated the high inter-rater reliability of assessing large or objective built environment features and the low inter-rater reliability of assessing small or subjective built environment features using Google Street View. New methods for auditing the built environment must be evaluated to understand if there are alternative tools through which researchers can assess all types of built environment features with high agreement. This paper investigates measures of inter-rater reliability of GigaPan®, a tool that assists with capturing high-definition panoramic images, relative to Google Street View.MethodsStreet segments (n = 614) in Pittsburgh, Pennsylvania in the United States were randomly selected to audit using GigaPan® and Google Street View. Each audit assessed features related to land use, traffic and safety, and public amenities. Inter-rater reliability statistics, including percent agreement, Cohen’s kappa, and the prevalence-adjusted bias-adjusted kappa (PABAK) were calculated for 106 street segments that were coded by two, different, human auditors.ResultsMost large-scale, objective features (e.g. bus stop presence or stop sign presence) demonstrated at least substantial inter-rater reliability for both methods, but significant differences emerged across finely detailed features (e.g. trash) and features at segment endpoints (e.g. sidewalk continuity). After adjusting for the effects of bias and prevalence, the inter-rater reliability estimates were consistently higher for almost all built environment features across GigaPan® and Google Street View.ConclusionGigaPan® is a reliable, alternative audit tool to Google Street View for studying the built environment. GigaPan® may be particularly well-suited for built environment projects with study settings in areas where Google Street View imagery is nonexistent or updated infrequently. The potential for enhanced, detailed imagery using GigaPan® will be most beneficial in studies in which current, time sensitive data are needed or microscale built environment features would be challenging to see in Google Street View. Furthermore, to better understand the effects of prevalence and bias in future reliability studies, researchers should consider using PABAK to supplement or expand upon Cohen’s kappa findings.

Trends in the use of automated algorithms for the detection of high-frequency oscillations associated with human epilepsy.

High-frequency oscillations (HFOs) in intracranial electroencephalography (EEG) are a promising biomarker of the epileptogenic zone and tool for surgical planning. Many studies have shown that a high rate of HFOs (number per minute) is correlated with the seizure-onset zone, and complete removal of HFO-generating brain regions has been associated with seizure-free outcome after surgery. In order to use HFOs as a biomarker, these transient events must first be detected in electrophysiological data. Because visual detection of HFOs is time-consuming and subject to low interrater reliability, many automated algorithms have been developed, and they are being used increasingly for such studies. However, there is little guidance on how to select an algorithm, implement it in a clinical setting, and validate the performance. Therefore, we aim to review automated HFO detection algorithms, focusing on conceptual similarities and differences between them. We summarize the standard steps for data pre-processing, as well as post-processing strategies for rejection of false-positive detections. We also detail four methods for algorithm testing and validation, and we describe the specific goal achieved by each one. We briefly review direct comparisons of automated algorithms applied to the same data set, emphasizing the importance of optimizing detection parameters. Then, to assess trends in the use of automated algorithms and their potential for use in clinical studies, we review evidence for the relationship between automatically detected HFOs and surgical outcome. We conclude with practical recommendations and propose standards for the selection, implementation, and validation of automated HFO-detection algorithms.

Pilot Study: Pneumatic Compression Garment Therapy for Postradiotherapy Laryngopharyngeal Edema.

Pneumatic compression garment therapy (PCGT) has been established as treatment for postradiotherapy lymphedema, and its use in head and neck patients is becoming more common. Although effects on interstitial edema of the cervical soft tissues have been studied, effects on internal laryngopharyngeal edema, as well as associated symptoms of dysphagia and dysphonia, have yet to be published. We surveyed 7 patients treated with radiation for head and neck cancer (HNC) who had also been prescribed PCGT for cervical lymphedema. Patients were asked about subjective experience with the device, and also administered the Eating Assessment Tool-10 (EAT-10) and Voice Handicap Index-10 (VHI-10) surveys regarding their symptoms after using PCGT. Laryngoscopy videos from these same periods were also reviewed and scored using a validated tool for assessing laryngopharyngeal edema. 85% of patients reported at least some improvement in dysphagia and dysphonia following PCGT. Average EAT-10 score after PCGT was 11.4 and average VHI-10 score after PCGT was 8.7. These compare more favorably to historical scores for the same questionnaires in similar patient populations. Laryngeal edema scores on endoscopic examination were not significantly different after at least 3 months of therapy (pre: 20.15, post: 20.21, P = .975); however, the utility of this result is limited by a low inter-rater reliability (Krippendorff α = .513). While we are unable to show any difference in objective assessment of laryngopharyngeal edema on endoscopic examination in this small pilot study, patients report substantial subjective improvement in postradiotherapy dysphagia and dysphonia following cervical PCGT that warrants more formal investigation.

The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability and challenges in its application

ObjectiveThe objective of the study is to assess the interrater reliability (IRR) and usability of the revised Cochrane risk of bias tool for randomized trials (RoB 2). Study Design and SettingThis is a cross-sectional study. Four raters independently applied RoB 2 on the primary outcome of a random sample of individually randomized parallel-group trials (randomized controlled trials (RCTs)). We calculated the Fleiss’ kappa for multiple raters, the time needed to complete the tool, and discussed the application of RoB 2 to identify difficulties and reasons for disagreement. ResultsA total of 70 outcomes from 70 RCTs were included. IRR was slight for overall judgment (IRR 0.16, 95% confidence interval (CI) 0.08–0.24); individual domain analysis gave IRR as moderate for “randomization process” (IRR 0.45, 95% CI 0.37–0.53), slight for “deviations from intended intervention” for RCTs assessing the effect of the assignment to an intervention (IRR 0.04, 95% CI −0.06 to 0.14), fair for those assessing the effect of adhering (IRR 0.21, 95% CI 0.11–0.31), and fair for the other domains, ranging from 0.22 (95% CI 0.14–0.30) for “missing outcome data” to 0.30 (95% CI 0.22–0.38) for “selection of reported results”. Mean time to apply the tool was 28 minutes (standard deviation 13.4) per study outcome. The main difficulties were due to poor knowledge of the subject matter of primary studies, new terminology, different approaches for some domains compared with the previous tool, and way of formulating signaling questions. ConclusionsRoB 2 is a detailed and comprehensive tool but difficult and demanding, even for raters with substantial expertise in systematic reviews. Calibration exercises and intensive training are needed before its application, to improve reliability.

Psychometric Testing of the Debriefing Assessment for Simulation in Healthcare (DASH) for Trainee-led, In Situ Simulations in the Pediatric Emergency Department Context.

Effective trainee-led debriefing after critical events in the pediatric emergency department has potential to improve patient care, but debriefing assessments for this context have not been developed. This study gathers preliminary validity and reliability evidence for the Debriefing Assessment for Simulation in Healthcare (DASH) as an assessment of trainee-led post-critical event debriefing. Eight fellows led teams in three simulated critical events, each followed by a video-recorded discussion of performance mimicking impromptu debriefings occurring after real clinical events. Three raters assessed the recorded debriefings using the DASH, and their feedback was collated. Data were analyzed using generalizability theory, Gwet's AC2, intraclass correlation coefficient (ICC), and coefficient alpha. Validity was examined using Messick's framework. The DASH instrument had relatively low traditional inter-rater reliability (Gwet's AC2= 0.24, single-rater ICC range= 0.16-0.35), with 30% fellow, 19% rater, and 23% rater by fellow variance. DASH generalizability (G) coefficient was 0.72, confirming inadequate reliability for research purposes. Decision (D) study results suggest the DASH can attain a G coefficient of 0.8 with five or more raters. Coefficient alpha was 0.95 for the DASH. A total of 90 and 40% of items from Elements 1 and 4, respectively, were deemed "not applicable" or left blank. Our results suggest that the DASH does not have sufficient validity and reliability to rigorously assess debriefing in the post-critical event environment but may be amenable to modification. Further development of the tool will be needed for optimal use in this context.

Reliability of the international spinal cord injury urodynamic basic data set version 1.0

A cross-sectional study. To assess interrater and intrarater reliability of the International Spinal Cord Injury (SCI) Urodynamic Basic Data Set (UBS) version 1.0. Urodynamic clinic at Maharaj Nakorn Chiang Mai Hospital. Two raters independently analyzed urodynamic tracings from 50 patients and completed the UBS twice, each test 1 month apart. The interrater and intrarater reliability of this data set were analyzed using Kappa, Weighted Kappa, and the Intraclass correlation coefficient (ICC). Of the 50 patients, 72% were male. The mean (SD) age was 48.2 (16.6) years. The median time (IQR) since the injury was 27 months (0-101 months). The interrater reliability of the items of UBS were substantial to almost perfect (0.78-0.99). The intrarater reliability of the first rater was fair to almost perfect (0.37-1.00). The intrarater reliability of the second rater was moderate to almost perfect (0.51-1.00). Relatively low interrater and intrarater reliability were observed in bladder compliance and urethral function items. The first version of UBS has acceptable interrater and intrarater reliability on most items. Although bladder compliance and urethral function have problematic interrater and intrarater reliability, they have been adjusted in the second version. Due to its simplicity and reliability, UBS is clinically useful for urodynamic assessment in people with SCI.

High Interrater Variability in Intraoperative Language Testing and Interpretation in Awake Brain Mapping Among Neurosurgeons or Neuropsychologists: An Emerging Need for Standardization

Brain mapping with direct electric stimulation is considered the gold standard for maximum safe resection of tumors affecting eloquent regions. However, no consensus exists in selection and interpretation of intraoperative testing for language and other cognitive domains. Our aim was to capture and statistically analyze variability in practices in intraoperative language testing among neurosurgeons and neuropsychologists in the United States, Europe, and the rest of the world. An electronic questionnaire was developed by a multidisciplinary team at Queen Square, London, and distributed internationally through selected organized societies. The survey included 2 domains: terminology and common understanding of clinical deficits; and selection of intraoperative tests used per specific brain region. Participants were stratified by specialty, years of experience, and monthly caseload. Data were analyzed using Krippendorff α, Wilcoxon rank sum test, and Kruskal-Wallis analysis of variance. A total of 137 specialists participated. A low agreement was recorded for each of the 20 questions (Krippendorff α=-0.023 to 0.312). Further subgroup analysis revealed low interrater reliability independent of specialism (neurosurgeons, α= 0.013-0.318 compared with nonneurosurgeons, α=-0.021 to 0.398; P= 0.808) and years of experience (<1 years, α=-0.003 to 0.282; 2-5 years, α= 0.009-0.327; 6-10 years, α= 0.003-0.234; and >10 years, α=-0.003 to 0.372; P= 0.200). The current study documents high interrater variability, regardless of specialism and years of experience in the cohort of neurosurgeons and language specialists surveyed and may be applicable to a wider group of specialists, indicating the need to reduce interobserver, interinstitutional and interspecialty variability, reach consensus, and increase the validity, interpretation, and predictive power of intraoperative mapping.

Determining the Agreement Between an Automated Respiratory Rate Counter and a Reference Standard for Detecting Symptoms of Pneumonia in Children: Protocol for a Cross-Sectional Study in Ethiopia.

BackgroundAcute respiratory infections (ARIs), primarily pneumonia, are the leading infectious cause of under-5 mortality worldwide. Manually counting respiratory rate (RR) for 60 seconds using an ARI timer is commonly practiced by community health workers to detect fast breathing, an important sign of pneumonia. However, correctly counting breaths manually and classifying the RR is challenging, often leading to inappropriate treatment. A potential solution is to introduce RR counters, which count and classify RR automatically.ObjectiveThis study aims to determine how the RR count of an Automated Respiratory Infection Diagnostic Aid (ARIDA) agrees with the count of an expert panel of pediatricians counting RR by reviewing a video of the child’s chest for 60 seconds (reference standard), for children aged younger than 5 years with cough and/or difficult breathing.MethodsA cross-sectional study aiming to enroll 290 children aged 0 to 59 months presenting to pediatric in- and outpatient departments at a teaching hospital in Addis Ababa, Ethiopia, was conducted. Enrollment occurred between April and May 2017. Once enrolled, children participated in at least one of three types of RR evaluations: (1) agreement—measure the RR count of an ARIDA in comparison with the reference standard, (2) consistency—measure the agreement between two ARIDA devices strapped to one child, and (3) RR fluctuation—measure RR count variability over time after ARIDA attachment as measured by a manual count. The agreement and consistency of expert clinicians (ECs) counting RR for the same child with the Mark 2 ARI timer for 60 seconds was also measured in comparison with the reference standard.ResultsPrimary outcomes were (1) mean difference between the ARIDA and reference standard RR count (agreement) and (2) mean difference between RR counts obtained by two ARIDA devices started simultaneously (consistency).ConclusionsStudy strengths included the design allowing for comparison between both ARIDA and the EC with the reference standard RR count. A limitation is that exactly the same set of breaths were not compared between ARIDA and the reference standard since ARIDA can take longer than 60 seconds to count RR. Also, manual RR counting, even when aided by a video of the child’s chest movements, is subject to human error and can result in low interrater reliability. Further work is needed to reach global consensus on the most appropriate reference standard and an acceptable level of agreement to provide ministries of health with evidence to make an informed decision on whether to scale up new automated RR counters.Trial RegistrationClinicalTrials.gov NCT03067558; https://clinicaltrials.gov/ct2/show/NCT03067558International Registered Report Identifier (IRRID)RR1-10.2196/16531

Low interrater reliability of brachial plexus MRI in chronic inflammatory neuropathies.

IntroductionMagnetic resonance imaging of the brachial plexus shows nerve thickening in approximately half of the patients with chronic inflammatory demyelinating polyneuropathy (CIDP) and multifocal motor neuropathy (MMN). The reliability of qualitative evaluation of brachial plexus MRI has not been studied previously.MethodsWe performed an interrater study in a retrospective cohort of 19 patients with CIDP, 17 patients with MMN, and 14 controls. The objective was to assess interrater variability between radiologists by using a predefined scoring system that allowed the distinction of no, possible, or definite nerve thickening.ResultsRaters agreed in 26 of 50 (52%) brachial plexus images; κ‐coefficient was 0.30 (SE 0.08, 95% confidence interval 0.14–0.46, P < .0005).DiscussionOur results provide evidence that interrater reliability of qualitative evaluation of brachial plexus MRI is low. Objective criteria for abnormality are required to optimize the diagnostic value of MRI for inflammatory neuropathies.

Repeat Auditing of Primary Health-care Facilities Against Standards for Occupational Health and Infection Control: A Study of Compliance and Reliability.

BackgroundThe elevated risk of occupational infection such as tuberculosis among health workers in many countries raises the question of whether the quality of occupational health and safety (OHS) and infection prevention and control (IPC) can be improved by auditing. The objectives of this study were to measure (1) audited compliance of primary health-care facilities in South Africa with national standards for OHS and IPC, (2) change in compliance at reaudit three years after baseline, and (3) the inter-rater reliability of the audit.MethodsThe study analyzed audits of 60 primary health-care facilities in the Western Cape Province of South Africa. Baseline external audits in the time period 2011–2012 were compared with follow-up internal audits in 2014–2015. Audits at 25 facilities that had both internal and external audits conducted in 2014/2015 were used to measure reliability.ResultsAt baseline, 25% of 60 facilities were “noncompliant” (audit score<50%), 48% “conditionally compliant” (score >50 < 80%), and only 27% “compliant” (score >80%). Overall, there was no significant improvement in compliance three years after baseline. Percentage agreement on specific items between internal and external audits ranged from 28% to 92% and kappa from -0.8 to 0.41 (poor to moderate).ConclusionLow baseline compliance with OHS–IPC measures and lack of improvement over three years reflect the difficulties of quality improvement in these domains. Low inter-rater reliability of the audit instrument undermines the audit process. Evidence-based investment of effort is required if repeat auditing is to contribute to occupational risk reduction for health workers.

Digital assessment of falls risk, frailty, and mobility impairment using wearable sensors

Falls are among the most frequent and costly population health issues, costing $50bn each year in the US. In current clinical practice, falls (and associated fall risk) are often self-reported after the “first fall”, delaying primary prevention of falls and development of targeted fall prevention interventions. Current methods for assessing falls risk can be subjective, inaccurate, have low inter-rater reliability, and do not address factors contributing to falls (poor balance, gait speed, transfers, turning). 8521 participants (72.7 ± 12.0 years, 5392 female) from six countries were assessed using a digital falls risk assessment protocol. Data consisted of wearable sensor data captured during the Timed Up and Go (TUG) test along with self-reported questionnaire data on falls risk factors, applied to previously trained and validated classifier models. We found that 25.8% of patients reported a fall in the previous 12 months, of the 74.6% of participants that had not reported a fall, 21.5% were found to have a high predicted risk of falls. Overall 26.2% of patients were predicted to be at high risk of falls. 29.8% of participants were found to have slow walking speed, while 19.8% had high gait variability and 17.5% had problems with transfers. We report an observational study of results obtained from a novel digital fall risk assessment protocol. This protocol is intended to support the early identification of older adults at risk of falls and inform the creation of appropriate personalized interventions to prevent falls. A population-based approach to management of falls using objective measures of falls risk and mobility impairment, may help reduce unnecessary outpatient and emergency department utilization by improving risk prediction and stratification, driving more patients towards clinical and community-based falls prevention activities.

Psychometric properties of the DSM-5 Clinician-Rated Dimensions of Psychosis Symptom Severity

ObjectiveThe Clinician-Rated Dimensions of Psychosis Symptom Severity (CRDPSS) was presented in the DSM-5 as a new scale to assess the dimensional aspects of psychosis in daily clinical practice. However, agreement in CRDPSS-ratings among raters in clinical practice remains unknown. We examined the inter-rater reliability (IRR) and convergent validity of the CRDPSS. MethodConsecutively recruited outpatients with recent onset schizophrenia spectrum disorders were included between January 2015 and July 2018. We collected multiple CRDPSS measurements of 335 participants, of whom 179 PANSS measurements were available. IRR was determined by comparing the CRDPSS-ratings of psychiatrists with a vis-à-vis contact and CRDPSS observations based on a detailed clinical presentation. IRR was expressed in Krippendorff's alpha and we estimated convergent validity by studying associations with PANSS factors by Spearman's rank correlation coefficient. ResultsInter-rater reliability scores measured in Krippendorff's alpha were low (0.35–0.64) for all items of the CRDPSS, except the item delusions (0.74). A three-factor model was found: ‘deficit/motor symptoms’, ‘positive symptoms’ and ‘mood symptoms’. Positive associations between CRDPSS factors with PANSS factors were found. ConclusionThis study demonstrated that the IRR of the CRDPSS between raters in clinical practice was insufficient. We did find some supporting evidence for convergent validity of the CRDPSS, but these results should be interpreted carefully due to low IRR. Consequently, general implementation in clinical practice should be done with caution and we recommend assessors to be trained.

44. DEVELOPMENT OF A TOOL FOR FACULTY TO ASSESS RESIDENT-LED LARGE GROUP TEACHING

Background Residency programs are required to develop residents as teachers. Much of the formal teaching by residents occurs in group settings; the existing published tools did not collect validity evidence for assessment of resident-led large group teaching. We aim to create a tool for faculty to assess resident teaching in this setting. Methods Initial content for the tool came from literature review and our personal experience leading resident-as-teacher curricula. Resident focus groups provided stakeholder input, informing the first round of tool revisions. A modified Delphi panel of 14 international faculty experts, over 2 rounds of revisions, provided feedback on the tool's elements. Anchors were designed and finalized after a third Delphi round. Study investigators piloted the tool with 10 video recordings of senior residents teaching from the 3 sites. Cronbach's alpha was calculated for internal consistency and intraclass correlation (ICC) for interrater reliability. Results The tool has 6 domains: learning climate, goals and objectives, content, promotion of understanding and retention, session management, and closure. Each domain contains 12 sub-elements which are described by 37 observable behaviors. The Cronbach's alpha was 0.88. The ICC was good or excellent for 13/37 sub-elements (35%), fair or poor for 22/37 sub-elements (59%) and the remaining 2 elements had no ICC score given no variability in rater scores. Conclusion A tool for faculty assessment of resident-led large group teaching was developed using robust methodology. In the pilot study, the assessed behaviors have good internal consistency, but low interrater reliability without rater training. In the next study phase, we will develop tool utilization standards, train faculty raters, and apply the tool to a larger video sample of resident teaching. We will collect validity evidence for the tool including ability to discriminate between novice and advanced teachers and its correlation with teaching milestones.

A Brief Measure of Interpersonal Interaction for 2-Player Serious Games: Questionnaire Validation.

BackgroundCompetitive and cooperative serious games have become increasingly popular in areas such as rehabilitation and education and have several potential advantages over single-player games. However, they are not suitable for everyone, and the user experience in competitive and cooperative serious games depends on many factors. One important factor is the verbal interaction between players, but the effect of this factor has not been extensively studied because of the lack of a validated measurement tool.ObjectiveThis paper aimed to validate a brief questionnaire that measures the verbal interaction between 2 players of a serious game. The questionnaire consists of 8 questions pertaining to the amount of conversation, its valence (positive or negative emotion), and its game relatedness.MethodsThe questionnaire was validated with 30 pairs of participants who played a competitive serious game for 10 min while being recorded with cameras. The questionnaire was filled out by both participants, an in-person observer, and 2 members of our research group who watched the videos. Results from these raters were used to develop questionnaire instructions, and the finalized questionnaire was given to 2 additional raters who were trained on 5 videos and then rated the other 25 videos independently.ResultsThe questionnaire’s interrater reliability is excellent for the amount of conversation and its game relatedness (intraclass correlation coefficients [ICCs] above 0.9). Interrater reliability is fair to good for conversation valence (ICCs between 0.4 and 0.7). We believe that the lower interrater reliability for valence is primarily because of a limited spread of valence values in our sample. Furthermore, questionnaire ratings were significantly correlated with players’ personality characteristics (eg, amount of conversation was correlated with extraversion) and pressure/tension experienced in the competitive game.ConclusionsThe validated questionnaire has the potential to be a useful tool for studying user experience in competitive and cooperative serious games. Furthermore, it could be adapted for other applications such as entertainment games. However, it has only been validated with unimpaired university students in a 2-player competitive serious game and should next be validated with different target populations (eg, stroke survivors) and different game designs (eg, cooperative games).

“Digitalizing Walkability”: Comparing Smartphone-Based and Web-Based Approaches to Measuring Neighborhood Walkability in Singapore

ABSTRACTWe evaluated two digitally enabled approaches to measuring neighborhood walkability: a smartphone-based, on-site pedestrian environmental audit tool, WalkTracker (WTracker), and remote, Web-based (Web) observations. Specifically, we examined street segments and intersections of a neighborhood in Singapore assessing: (1) the Inter-Rater Reliability (IRR) of each approach; (2) the Inter-Method Reliability (IMR) across the two approaches; and (3) the average observation times of the two approaches. Each approach had high IRR for the land use and traffic-related domains, with the Web performing better than WTracker for land use. In these same two domains, the two tools were relatively consistent (high IMR), although higher agreement was found within the tools than across them (IRR higher than IMR). For subjective or fine-grained features, both approaches had low IRR, with the Web-based approach performing worse than the app-based approach. Performance across the instruments was also worse than the reliability of measurements within each instrument (IMR lower than IRR). Some items were not observable via the Web. In terms of observation time, there was no statistically significant time difference in measurements between the two observation methods, not including the round-trip travel time to the site. A hybrid approach, combining the two approaches, might be most appropriate.