High Levels Of Inter-rater Reliability Research Articles

Sensitivity, Specificity, and Reliability of the CTS-6 for Carpal Tunnel Syndrome Administered by Medical Assistants

The CTS-6 can be used clinically to diagnose carpal tunnel syndrome (CTS) and has demonstrated high levels of interrater reliability when administered by nonexpert clinicians. Our purpose was to assess sensitivity (Sn), specificity (Sp), and interrater reliability of the CTS-6 when administered by medical assistants (MAs). A series of patients presenting to an academic, upper-extremity surgery clinic were screened using CTS-6 between May and June of 2023. The CTS-6 was first administered by one of seven MAs and then by one of four fellowship-trained upper-extremity surgeons. In addition to recording baseline demographics, the results of each of the six history and examination components of the CTS-6 were recorded, as was the cumulative CTS-6 score (0-26). Surgeons were blinded to the scores obtained by the MAs. Interrater reliability (Cohen's kappa) was determined between the groups with respect to the diagnosis of CTS and the individual CTS-6 items. Sensitivity/specificity was calculated for the MA-administered CTS-6, using the score obtained by the surgeon as the reference standard. A CTS-6 score >12 was considered diagnostic of CTS. Two hundred eighteen patients were included, and 26% had a diagnosis of CTS. The MA group demonstrated a Sn/Sp of 84%/91% for the diagnosis of CTS. Interrater reliability was substantial (Cohen's kappa: 0.72, 95% confidence interval: 0.62-0.83) for MAs compared with hand surgeons for the diagnosis of CTS. For individual CTS-6 components, agreement was lowest for the assessment of two-point discrimination (fair) and highest for the assessment of subjective numbness (near perfect). The CTS-6 demonstrates substantial reliability and high Sn/Sp when administrated by MAs in an upper-extremity clinic. These data may be used to inform the development of CTS screening programs and future investigations in the primary care setting. Diagnostic II.

Assessing Creativity in the Language Classroom with the Consensual Assessment Technique

ABSTRACT In this study, The Consensual Assessment Technique is used to measure the creativity of 100 teaching artifacts from world language classrooms. These teaching artifacts were self-selected by K-12 world language teachers and categorized as creative or uncreative. The first set of artifacts was collected prior to the COVID-19 pandemic, in spring of 2019; the second set of artifacts was collected during the COVID-19 pandemic, in the winter of 2020, after several months of online teaching. The two sets of artifacts were randomly mixed and content experts rated the creativity of the 100 total artifacts. High levels of inter-rater reliability were obtained. Results indicated that creativity was not impacted by the move to online instruction during the pandemic and that use of target language (e.g. Spanish, French) may have a role to play in creativity.

Interprofessional Gratitude: Partnering With Registered Nurses as Embedded Participants in a Medical Education Patient Death Simulation Experience

BackgroundEmbedded participants (EPs) are non-learners trained to specific roles in simulation encounters. EPs contributing to specialized simulations may require unique skills. Partnering with health care providers with competencies complementary to desired EP roles is a possible way to address this need. Our team partnered with registered nurses (RNs) as EPs in a simulation in which the objective was for medical students to experience a imulated death in a supported environment. In this qualitative descriptive study, we address: What are medical students’ reactions to experienced RNs partnering as EPs during a simulation about the death of a patient? MethodsTwo hundred seventy-seven preclinical, second-year medical students wrote written reflections that were thematically analyzed with high levels of interrater reliability. ResultsStudents expressed gratitude for RNs partnering as EPs, described by subthemes of gratitude for nursing leadership, team-based medical care, and processing opportunities. ConclusionsRNs partnering as EPs effectively guided medical students through a patient death simulation. Simulationists should seek ways to leverage the deep, varied skill set of experienced RNs to partner as EPs in specialized medical education simulation.

Pedestrian street behavior mapping using unmanned aerial vehicles. A case study in Santiago de Chile.

Objective observation of pedestrian behavior on the street has traditionally been difficult due to intensive commitment of time and resources with spatial analysis of pedestrian locations encountering additional problems. Recently, Unmanned Aerial Vehicles (UAVs) have gained popularity due to the significant improvements they offer over other conventional observation systems, such as their ability to cover larger surface areas in less time. This study tests the performance of UAV-based observation techniques in measuring pedestrian activity in two comparative settings in Santiago de Chile. The study develops an alternative technique adapting the behavioral mapping methodology that allows acquiring information about the people's activities and the places where they are carried out. In this study a set of streets in the city of Santiago de Chile was selected as a case study, and the reliability of those observations was tested among raters in a population sample. Further, the use of a Geographic Information System (GIS) in the data coding process is detailed and exemplified using some of its spatial analysis tools. The results show high levels of inter-rater reliability in the different categories of recorded data. Finally, we discuss the advantages and limitations in observing pedestrian behavior using this technology and observation technique.

Use of Ketamine, Propofol and Their Combination (KETOFOL) For Procedural Sedation in Emergency Department: A Review

For Procedural Sedation, sedative and Analgesic agents are frequently used in Emergency Department but titration of anesthetic doses should be performed with care, and patients should be continuously monitored. The use of Ketamine, Propofol and their combination (Ketofol) is in common practice, but there is currently no intravenous anesthetic agent that is ideal. Therefore, this review was conducted to analyze the efficacy as well as the potential side effects of these anesthetic agents during procedural sedation. Method: For this purpose, Medline, EMBASE, CCRCT and CINAHL were searched and systematically analyzed and the meta-analysis included all English-language randomized control trials (RCTs) comparing K-P vs propofol for procedural sedation in ED. The study included the data of ED patients who received procedural sedation for non-elective unpleasant procedures. Results: As a result of the removal of duplicate citations and studies that did not fulfill eligibility requirements, a total of 06 RCTs involving 932 patients (412 in the propofol group and 520 in the K-P group). Very high levels of inter-rater reliability (j = 0.88; 95% CI = 0.68 to 1.0) were found in the final selection of included trials, with 95.2% agreement. Conclusion: Data revealed that the combination of Ketamine with propofol was the most effective anesthetic combination in the larger randomized, prospective studies carried out in the ED that had sufficient power to use the maintenance of vital signs and the success of the procedure as endpoints are still required

Early Identification of Children at Risk of Communication Disorders: Introducing a Novel Battery of Dynamic Assessments for Infants.

Many children with communication disorders (CDs) experience lengthy gaps between parental reporting of concerns and formal identification by professionals. This means that children with CDs are denied access to early interventions that may help support the development of communication skills and prevent possible negative sequelae associated with long-term outcomes. This may be due, in part, to the lack of assessment instruments available for children younger than 3 years of age. This study therefore reports on promising preliminary data from a novel set of valid dynamic assessment (DA) measures designed for infants. We recruited 53 low-risk children and two groups of children considered to be at high risk for CDs (n = 17, social high risk, and n = 22, language high risk) due to family members with language and social communication difficulties. The children were between 1 and 2 years of age and were assessed using a battery of five DA tasks related to receptive vocabulary, motor imitation, response to joint attention, turn taking, and social requesting. A set of standardized measures were also used. The DA tasks showed high levels of interrater reliability and relationships with age across a cross-sectional sample of children from the low-risk group. Three tasks showed moderate to strong correlations with standardized measures taken at the same age, with particularly strong correlations between the DA of receptive vocabulary and other receptive language measures. The DA of receptive vocabulary was also the only task to discriminate between the three risk groups, with the social high risk group scoring lower. These results provide preliminary information about early DA tasks, forming the basis for further research into their utility. DA tasks might eventually facilitate the development of new methods for detecting CDs in very young children, allowing earlier intervention and support.

Coding social interactions in naturalistic settings: The taxonomy of dyadic conversation.

Social interaction and conversation is an essential aspect of human behavior, yet existing methods for coding conversations are outdated, and often can only be used in contrived research settings. The Taxonomy of Dyadic Conversation (TDC) is a coding system designed to code dyadic interactions in natural settings by labeling the utterances and turns taken within an interaction using speech categories. The TDC was used to code child-caregiver and adult-adult conversations in a children's museum and during a public forum, respectively. Results supported hypotheses that predicted adult-adult interactions would contain more Declarative Statement and Acknowledgment utterances than child-caregiver interactions, while child-caregiver interactions contained fewer Conversational Turns, as well as more Command and Encouragement utterances. Results also indicated high levels of inter-rater reliability. The potential for additions and modifications to be applied to the standard TDC is discussed.

An Evaluation of KELVIN, an Artificial Intelligence Platform, as an Objective Assessment of the MDS UPDRS Part III.

Parkinson's disease severity is typically measured using the Movement Disorder Society Unified Parkinson's disease rating scale (MDS-UPDRS). While training for this scale exists, users may vary in how they score a patient with the consequence of intra-rater and inter-rater variability. In this study we explored the consistency of an artificial intelligence platform compared with traditional clinical scoring in the assessment of motor severity in PD. Twenty-two PD patients underwent simultaneous MDS-UPDRS scoring by two experienced MDS-UPDRS raters and the two sets of accompanying video footage were also scored by an artificial intelligence video analysis platform known as KELVIN. KELVIN was able to produce a summary score for 7 MDS-UPDRS part 3 items with good inter-rater reliability (Intraclass Correlation Coefficient (ICC) 0.80 in the OFF-medication state, ICC 0.73 in the ON-medication state). Clinician scores had exceptionally high levels of inter-rater reliability in both the OFF (0.99) and ON (0.94) medication conditions (possibly reflecting the highly experienced team). There was an ICC of 0.84 in the OFF-medication state and 0.31 in the ON-medication state between the mean Clinician and mean Kelvin scores for the equivalent 7 motor items, possibly due to dyskinesia impacting on the KELVIN scores. We conclude that KELVIN may prove useful in the capture and scoring of multiple items of MDS-UPDRS part 3 with levels of consistency not far short of that achieved by experienced MDS-UPDRS clinical raters, and is worthy of further investigation.

Remarkable recoveries: an interpretation of recovery narratives using the CHIME model

PurposeThe purpose of this paper is to review the 16 published non-student Recovery Heroes and Remarkable Lives accounts published in Mental Health and Social Inclusion, using the connection, hope, identity, meaning and empowerment (CHIME) framework.Design/methodology/approachAll 16 accounts were rated independently by four researchers and evaluated in terms of whether each account met the five criteria for the CHIME model.FindingsAll accounts met the criteria for the CHIME model, with the exception of one, which still met four of the five criteria. Evidence was presented which suggests that the model can be extended to creativity, connectedness, hope, identity, meaning and empowerment (C-CHIME), to incorporate creativity.Research limitations/implicationsWhile a certain level of subjectivity is required in deciding how each account meets the CHIME criteria, there were high levels of inter-rater reliability within the research team. Creativity had a central place in all the accounts.Practical implicationsThe revised C-CHIME model can be used by practitioners to examine accounts of recovery in a more focussed manner and may also help in devising recovery action plans.Social implicationsThe recovery model privileges both professional and lived experience perspectives on recovery. The current review highlights how much we can benefit from the wisdom contained in first person accounts.Originality/valueThis review adds to the existing literature and highlights the importance of creativity for mental health recovery.

Assessing inter-rater reproducibility in MScanFit MUNE in a 6-subject, 12-rater “Round Robin” setup

ObjectiveTo assess the inter-rater reliability of MScanFit MUNE using a “Round Robin” research design. MethodsTwelve raters from different centres examined six healthy study participants over two days. Median, ulnar and common peroneal nerves were stimulated, and compound muscle action potential (CMAP)-scans were recorded from abductor pollicis brevis (APB), abductor digiti minimi (ADM) and anterior tibial (TA) muscles respectively. From this we calculated the Motor Unit Number Estimation (MUNE) and “A50”, a motor unit size parameter. As statistical analysis we used the measures Limits of Agreement (LOA) and Coefficient of Variation (COV). Study participants scored their perception of pain from the examinations on a rating scale from 0 (no pain) to 10 (unbearable pain). ResultsBefore this study, 41.6% of the raters had performed MScanFit less than five times. The mean MUNE-values were: 99.6 (APB), 131.4 (ADM) and 126.2 (TA), with LOA: 19.5 (APB), 29.8 (ADM) and 20.7 (TA), and COV: 13.4 (APB), 6.3 (ADM) and 5.6 (TA). MUNE-values correlated to CMAP max amplitudes (R2-values were: 0.463 (APB) (p<0.001), 0.421 (ADM) (p<0.001) and 0.645 (TA) (p<0.001)). The average perception of pain was 4. DiscussionMScanFit indicates a high level of inter-rater reliability, even with only limited rater experience and is overall reasonably well tolerated by patients. These results may indicate MScanFit as a reliable MUNE method with potential as a biomarker in drug trials.

Beliefs about Social Distancing During COVID-19 Stay-At-Home Orders: A Theory-Based Salient Belief Elicitation

Understanding the beliefs about social distancing behaviors is required to inform 2019 coronavirus disease (COVID-19) interventions that are based on theory, research, and evidence. This study investigated the salient beliefs related to social distancing. United States adults (n = 106) recruited from Amazon’s Mechanical Turk completed an online reasoned action approach belief elicitation from April 19 to April 25, 2020. Behavioral beliefs (advantages and disadvantages), normative beliefs (approvers and disapprovers), and control beliefs (facilitators and barriers) related to social distancing were elicited via open-ended questions. A content analysis was performed, and kappa statistics revealed high levels of interrater reliability (α = 0.86-0.96). Results revealed that a perceived salient advantage to social distancing was individual COVID-19 prevention, more so than community prevention. The most cited disadvantage was that social distancing could prevent participants from socially interacting with others, which could negatively impact mental health. Family and friends were the most mentioned approvers, while people who hold conservative ideologies and negative attitudes about COVID-19 were the most frequent disapprovers. Supply accessibility and store policies were the most listed facilitators. Results suggest three implications. First, pandemic-related public health and social marketing campaigns should focus more on individual health benefits than community health benefits. Second, digital public health interventions that address social connectedness and mental health outcomes are critical during pandemics. Third, public health scientists and practitioners should work with local and national media outlets and political leaders to create community-tailored and evidence-based information to increase adherence of mitigation strategies.

A reliable interictal EEG grading scale for children with infantile spasms – The 2021 BASED score

ObjectiveTo develop an improved interictal electroencephalogram (EEG) grading scale for children with infantile spasms founded on elements with adequate inter-rater reliability (IRR) to justify its further study for clinical and research purposes. MethodsThree blinded reviewers assessed five-minute sleep epochs in 93 EEGs from 62 children (31 consecutive controls, 31 consecutive infantile spasms [pretreatment and posttreatment studies]) using a longitudinal bipolar montage. We determined the IRR of background amplitude, epileptiform discharges, >3 spike foci (including <50 % or >50 %), grouped multifocal spikes, paroxysmal voltage attenuations, and symmetry of sleep spindles. Data were used to finalize the 2021 BASED (Burden of AmplitudeS and Epileptiform Discharges) score. ResultsAll elements included in the 2021 BASED score had moderate to near perfect IRR. Among controls, >200 μv background waves occurred commonly in the bilateral posterior temporal (T3-T5, T4-T6) and midline (Fz-Cz, Cz-Pz) regions. Excluding midline and occipital channels (which have normal high amplitude background waves), we designated abnormal high amplitude background waves as >200 μv for most channels, but >300 μv for T3-T5 and T4-T6. The IRR was moderate to near perfect for <50 % >3 spike foci, >50 % >3 spike foci, paroxysmal voltage attenuations, grouped multifocal spikes (GMFS), and symmetric sleep spindles. Paroxysmal voltage attenuations, GMFS, and >50 % >3 spike foci all significantly distinguished pretreatment from posttreatment studies whereas symmetric sleep spindles did not (as planned, the latter was not included in the 2021 BASED score). When the 2021 BASED score was applied to the 22 children with infantile spasms achieving clinical remission with treatment, 19 met criteria for electroclinical remission and three did not. SignificanceThe 2021 BASED score includes elements with high levels of IRR and correlates well with the presence or absence of infantile spasms.

Establishing a clinical high-risk program in Tunisia, North Africa: A pilot study in early detection and identification.

Early identification and prevention research has provided huge advances in our understanding of early screening and identification of young people at clinical high-risk (CHR). Most of these procedures were developed in high-income countries, yet middle-income countries in North Africa such as Tunisia can benefit from these empirically-based assessment approaches. Using established procedures, nine Tunisian psychiatric raters were trained on structured assessments: the CAARMS, BPRS, and SCID to high standards of interrater reliability. These raters developed a clinical high-risk program (CHiRP) in Tunisia and recruited 10 patients who were exhibiting possible signs of CHR. These patients were evaluated to determine if they met criteria for a CHR group, such as Attenuated Psychosis. Trained raters met the following interrater reliability criteria for the CAARMS and BPRS (ICC=.80 or greater) and the SCID (Kappa=.75 or greater). Of 10 pilot patients, six were classified as CHR and belonging to the Attenuated Psychosis Group. One of the six patients converted to psychosis 3months after study entry for a conversion rate of 17% which is comparable with currently published rates globally. The first CHR program has been established in Tunisia, a middle-income country using methods developed in high income countries. Efforts aimed at assembling a group of prevention-oriented psychiatrists, obtaining administrative support, and training raters to high levels of interrater reliability were successful. The feasibility was demonstrated for screening, assessing, treating, and following-up of 10 CHR patients suggesting that conversion rates are comparable to those of Western and European countries.

Psychometric Properties of the General Organizational Index (GOI): A Measure of Individualization and Quality Improvement to Complement Program Fidelity

To assess the implementation of effective practices, mental health programs need standardized measures. The General Organizational Index (GOI), although widely used for this purpose, has received minimal psychometric research. For this study, we assessed psychometric properties of the GOI scale administered four times over 18 months during the implementation of a new program in 11 sites. The GOI scale demonstrated high levels of interrater reliability (.97), agreement between assessors on item ratings (86% overall), internal consistency (.77–.80 at three time points), sensitivity to change, and feasibility. We conclude that the GOI scale has acceptable psychometric properties, and its use may enhance implementation and research on evidence-based mental health practices.Trial registration: REK2015/2169. ClinicalTrials.gov Identifier: NCT03271242

A Two-Stage Method for Obtaining Reliable Teacher Assessments of Writing

In many countries, systems have employed externally imposed large-scale standardised assessments in seeking to obtain reliable and comparable assessments. To attain the same objectives while placing value on professional judgement, appropriate methods of assessment for classroom teachers need to be developed and tested. Accordingly, this paper examines the level of reliability of assessments made by classroom teachers of narrative writing using a two-stage classroom assessment method. The students involved in the study are primary students. The results show high levels of inter-rater reliability among teachers. The findings reproduce and expand on previous evidence indicating the two-stage method is a viable method of classroom teacher assessment of written performances. Implications of the results for assessment practices in education are discussed in light of background literature, with a focus on the widely expressed desire to value professional expertise.

Reliable application of the MATH taxonomy sheds light on assessment practices

Abstract The MATH taxonomy classifies questions according to the mathematical skills required to answer them. It was created to aid the development of more balanced assessments in undergraduate mathematics and has since been used to compare different assessment regimes across school and university. To date, there has been no systematic investigation of the reliability of the taxonomy when applied by multiple coders, and it has only been applied in a limited range of contexts. In this paper, we outline a calibration process which enabled four novice coders to attain a high level of inter-rater reliability. In addition, we report on the results of applying the taxonomy to different secondary school exams and to all assessment questions in a first-year university mathematics module. The results confirm previous findings that there is a difference between the mix of skills assessed in school and university mathematics exams, although we find a notably different assessment profile in the university module than in previous work. The calibration process we describe has the potential to be used more widely, enabling reliable use of the MATH taxonomy to give insight into assessment practices.

Validity and reliability of the Persian version of the Quick Exposure Check (QEC) in Iranian construction workers.

The Quick Exposure Check (QEC) is an instrument to identify occupational risk factors, and to screen subjects for the risk of musculoskeletal disorders in their workplace. The aim of this prospective observational study was the cross-cultural adaptation of the QEC for Persian (QEC-Pr) speaking workers and the determination of the minimally required psychometric properties, including reliability and validity, in a Persian population. The forward-backward translation was completed according to the published guidelines. A total of 217 male construction workers were recruited. The Rapid Upper Limb Assessment (RULA) tool was used as the criterion. The Inter- and intra-rater reliability was performed respectively in a sub-sample of n = 50 and n = 30. Convergent validity was determined from the correlation between the QEC-Pr and the RULA. Internal consistency was assessed using Cronbach's α. High levels of inter-rater reliability (ICC2.1 range = 0.79-0.93), intra-rater reliability (ICC2.1 range = 0.74-0.89) and internal consistency (α= 0.74) were obtained. Concurrent validity between the different sections of the QEC and RULA scores were determined to be fair but below the desired level (range r = 0.41-0.44). The QEC-Pr can be applied as a specific risk assessment instrument for different occupational situations, such as the construction industry, in Persian language populations.

Mathematics Graduate Student Instructor Observation Protocol (GSIOP): Development and Validation Study

This paper presents the development and validation of the 17-item mathematics Graduate Student Instructor Observation Protocol (GSIOP) at two universities. The development of this instrument attended to some unique needs of novice undergraduate mathematics instructors while building on an existing instrument that focused on classroom interactions particularly relevant for students’ development of conceptual understanding, called the Mathematical Classroom Observation Protocol for Practices (MCOP2). Instrument validation involved content input from mathematics education researchers and upper-level mathematics graduate student instructors at two universities, internal consistency analysis, interrater reliability analysis, and structure analyses via scree plot analysis and exploratory factor analysis. A Cronbach-Alpha level of 0.868 illustrated a viable level for internal consistency. Crosstabulation and correlations illustrate high level of interrater reliability for all but one item, and high levels across all subsections. Collaborating a scree plot with the exploratory factor analysis illustrated three critical groupings aligning with the factors from the MCOP2 (student engagement and teacher facilitation) while adding a third factor, lesson design practices. Taken collectively, these results indicate that the GSIOP measures the degree to which instructors’ and students’ actions in undergraduate mathematics classrooms align with practices recommended by the Mathematical Association of America (MAA) using a three-factor structure of teacher facilitation, student engagement, and design practices.

LATINO HOSPICE PATIENTS’ INVOLVEMENT IN CARE DECISIONS AND PLANNING: FACTORS THAT SHAPE PARTICIPATION PREFERENCES

Patient participation in care decisions is a primary tenet of patient-centered care and has been emphasized in policies and programs shaping clinical practice. Patient self-determination and involvement is associated with improved outcomes, including patient satisfaction and compliance. However, studies exploring Latino patients’ care involvement preferences find a cultural preference for limited autonomy and shared or family-based decision-making. This study describes terminally-ill older Latinos and their proxies’ preferences regarding involvement in decision-making and care planning following a referral to hospice care. Semi-structured interviews were conducted with 13 hospice-enrolled Latinos 65 or older, or their proxies. The sample is predominantly female (54% of all patients; 80% of proxies) and Mexican (77%). The average age of all patients was 82 years, and of proxies 54. Patients interviewed had an average of 7 years of education and proxies an average of 11 years. Patients were receiving care in a variety of settings, including inpatient hospice units, nursing homes, and community. Interviews were transcribed verbatim in the original language and coded thematically using an inductive approach. A second coder was used to code a Spanish and an English transcript (15% of the data), and a high level of interrater reliability was obtained (k = 0.85). Findings elucidate the paths through which culture, religion, and trust in provider shape preferences for limited autonomy and planning, confirming previous findings. However, this was not replaced by a preference for family-based decision-making. Findings highlight the importance of Latinos having access to culturally-competent primary care providers and fostering lasting relationships.

A Two‐Stage Method for Classroom Assessments of Essay Writing

AbstractTo capitalize on professional expertise in educational assessment, it is desirable to develop and test methods of rater‐mediated assessment that enable classroom teachers to make reliable and informative judgments. Accordingly, this article investigates the reliability of a two‐stage method used by classroom teachers to assess primary school students’ persuasive writing. Stage 1 involves pairwise comparisons and stage 2 involves rating against calibrated exemplars from stage 1 plus performance descriptors. A high level of interrater reliability among teachers was obtained. This is consistent with previous evidence that the two‐stage method is a viable classroom assessment method without extensive training and moderation. Implications for assessment practices in education are discussed with a focus on the widely expressed desire to value professional expertise.