Inter-rater Reliability Study Research Articles

Field testing two existing, standardized respiratory severity scores (LIBSS and ReSViNET) in infants presenting with acute respiratory illness to tertiary hospitals in Rwanda - a validation and inter-rater reliability study.

There is a substantial burden of respiratory disease in infants in the sub-Saharan Africa region. Many health care providers (HCPs) that initially receive infants with respiratory distress may not be adequately skilled to differentiate between mild, moderate and severe respiratory symptoms, which may contribute to poor management and outcome. Therefore, respiratory severity scores have the potential to contributing to address this gap. to field-test the use of two existing standardized bronchiolitis severity scores (LIBSS and ReSViNET) in a population of Rwandan infants (1-12 months) presenting with respiratory illnesses to urban, tertiary, pediatric hospitals and to assess the severity of respiratory distress in these infants and the treatments used. A cross-sectional, validation study, was conducted in four tertiary hospitals in Rwanda. Infants presenting with difficulty in breathing were included. The LIBSS and ReSViNET scores were independently employed by nurses and residents to assess the severity of disease in each infant. 100 infants were recruited with a mean age of seven months. Infants presented with pneumonia (n = 51), bronchiolitis (n = 36) and other infectious respiratory illnesses (n = 13). Thirty-three infants had severe disease and survival was 94% using nurse applied LIBSS. Regarding inter-rater reliability, the intra-class correlation coefficient (ICC) for LIBSS and ReSViNET between nurses and residents was 0.985 (95% CI: 0.98-0.99) and 0.980 (0.97-0.99). The convergent validity (Pearson's correlation) between LIBSS and ReSViNET for nurses and residents was R = 0.836 (p<0.001) and R = 0.815 (p<0.001). The area under the Receiver Operator Curve (aROC) for admission to PICU or HDU was 0.956 (CI: 0.92-0.99, p<0.001) and 0.880 (CI: 0.80-0.96, p<0.001) for nurse completed LIBSS and ReSViNET respectively. LIBSS and ReSViNET were designed for infants with bronchiolitis in resource-rich settings. Both LIBSS and ReSViNET demonstrated good reliability and validity results, in this cohort of patients presenting to tertiary level hospitals. This early data demonstrate that these two scores have the potential to be used in conjunction with clinical reasoning to identify infants at increased risk of clinical deterioration and allow timely admission, treatment escalation and therefore support resource allocation in Rwanda.

A Systematic Review of Clinical Prediction Rules for the Diagnosis of Influenza.

Clinical prediction rules (CPRs) can assist clinicians by focusing their clinical evaluation on the most important signs and symptoms, and if used properly can reduce the need for diagnostic testing. This study aims to perform an updated systematic review of clinical prediction rules and classification and regression tree (CART) models for the diagnosis of influenza. We searched PubMed, CINAHL, and EMBASE databases. We identified prospective studies of patients presenting with suspected influenza or respiratory infection and that reported a CPR in the form of a risk score or CART-based algorithm. Studies had to report at a minimum the percentage of patients in each risk group with influenza. Studies were evaluated for inclusion and data were extracted by reviewers working in parallel. Accuracy was summarized descriptively; where not reported by the authors the area under the receiver operating characteristic curve (AUROCC), predictive values, and likelihood ratios were calculated. We identified 10 studies that presented 14 CPRs. The most commonly included predictor variables were cough, fever, chills and/or sweats, myalgias, and acute onset, all which can be ascertained by phone or telehealth visit. Most CPRs had an AUROCC between 0.7 and 0.8, indicating good discrimination. However, only 1 rule has undergone prospective external validation, with limited success. Data reporting by the original studies was in some cases inadequate to determine measures of accuracy. Well-designed validation studies, studies of interrater reliability between telehealth an in-person assessment, and studies using novel data mining and artificial intelligence strategies are needed to improve diagnosis of this common and important infection.

Novices versus experts: a blinded study of inter-rater reliability of sport-related concussion video signs in the national rugby league

Background: Recent consensus definitions were developed of video signs to identify possible sport-related concussions (SRC). The reliability of these definitions is yet to be evaluated. The primary aim of this study was to determine the inter-rater reliability between novice and expert clinicians in recognising observable signs of SRC.

First step in assessment of VipGrade®, a computerized clinical decision system to assess Vipera envenomation grading: a single-center interrater reliability study

Methods We conducted a retrospective review of Vipera spp. snakebite cases registered by the PCC of Bordeaux, France, between January 1, 2018, and December 31, 2020, evaluating the agreement between VipGrade® assessments, toxicologists’ assessments, and current guidelines. Results 133 patients with Vipera aspis snakebites were included. There was 100% agreement in severity grading by PCC guidelines and VipGrade®. However, grading by toxicologists and VipGrade® diverged in 19 cases (85% agreement; κ = 0.80; 95% CI: 0.71 to 0.87). Conclusions The VipGrade® tool’s grading reflects current PCC guidelines, which are authoritative in France, and may allow for a more rapid and standardized determination of management and follow-up of viper-bitten patients. It should be noted, however, that the more complex and dynamic aspects of management are not included in VipGrade®. Its purpose is to supplement, not replace, the advice of the PCC’s clinical toxicologists, and this advice should be sought whenever a viper bite is encountered in clinical practice.

The Intra- and Inter-Rater Reliability of a Hip Rotation Range-of-Motion Measurement Using a Smartphone Application in Academy Football (Soccer) Players.

The clinical assessment of hip rotation range-of-motion (ROM) is important for managing hip and groin injuries in footballers. Previously published reliability studies on hip ROM have employed protocols that are difficult to replicate under everyday clinical conditions. This single trial, intra- and inter-rater reliability study included 41 male academy football (soccer) players, aged 14–15 years, from one European football academy. Passive hip internal rotation (IR) and external rotation (ER) ROM were measured in supine with hip and knee flexed to 90°. The ROM was determined using a smartphone application, with the smartphone attached to the lower leg. The tests were performed on two separate occasions, one week apart, by two different physiotherapists and on both sides (left and right hips). Reliability was evaluated using Intra-Class Correlation Coefficients (ICCs) and Minimal Detectable Change (MDC). Hip IR and ER ROM displayed moderate to good intra-rater agreement (ICCs 0.54–0.75), with MDCs ranging from 10.9° to 16.4°. Inter-rater reliability displayed poor to moderate reliability (ICCs 0.33–0.75), with MDCs ranging from 11.7° to 16.5°. A hip rotation ROM test using a smartphone application and a protocol closely reflecting everyday clinical conditions displayed moderate to good intra-rater reliability and poor to moderate inter-rater reliability. Due to the high MDCs, the practical applicability of this test procedure is limited and further refinement is necessary.

The reliability of the scapular posture and scapular dyskinesis tests in rugby union players.

Impact forces during rugby can contribute to scapula dysfunction and shoulder pain. Reliable assessment of static and dynamic scapula position is paramount in managing athletes with, and those at risk of developing, shoulder pain. To determine the reliability of static scapular posture (SP), clavicular tilt angle (CTA) and the scapular dyskinesis (SD) assessments by expert and student therapists. The study design was an inter-rater and intra-rater reliability study in male university level rugby union players. Four sport rehabilitation students and one experienced physiotherapist evaluated the position of the scapular and clavicle of male university-level rugby union players (inter-rater participants: session 1: n=17, session 2: n=12 and session 3: n=16; (intra-rater participants: n = 12). Participants attended 3 testing sessions, each 1 week apart. Scapular orientation and motion were assessed in five planes of movement and using the Scapular Dyskinesis Test (SDT) respectively. The inter-rater analysis included all participants from each session, while in the intra-rater analysis included only the 12 participants who attended all three testing sessions. Kappa coefficient values and percentage agreement ratings for students compared to the experienced therapist were: SP=poor to fair (-0.01 to 0.33), (27% - 94%); SDT=slight (0.16; 41%); CTA=fair (0.21; 59%). Test-retest (intra-rater) agreement was fair to moderate (0.22 - 0.44; 69% - 95%), slight (0.12; 47%), and fair (0.39; 77%) for the SP, SDT, and CTA, respectively. Static and dynamic evaluation of the shoulder by students and an experienced therapist has poor to moderate reliability and should not be used to make clinical decisions based on observation alone.

An interrater reliability study on the Gothenburg obstetric triage system- a new obstetric triage system

BackgroundTriage, identifying patients with critical and time-sensitive disorders, is an integrated process in general emergency medicine. Obstetric triage is more specialised, requiring assessment of both woman, fetus and labour status. Failure to identify severely ill obstetric patients has repeatedly led to maternal morbidity and mortality. Reliable triage systems, adapted to obstetric patients as well as local conditions, are thus essential. The study aims to assess the interrater reliability (IRR) of the Gothenburg Obstetric Triage System (GOTS).MethodsMidwives (n = 6) and registered nurses with no experience in managing obstetric patients (n = 7), assessed 30 paper cases based on actual real-life cases, using the GOTS. Furthermore, a reference group consisting of two midwives and two obstetricians, with extensive experience in obstetric care, determined the correct triage level in order to enable analysis of over- and undertriage. IRR was assessed, both with percentage of absolute agreement and with intra-class correlation coefficients (ICC) with 95% confidence intervals (CI).ResultsA total of 388 assessments were performed, comprising all five levels of acuity in the GOTS. Absolute agreement was found in 69.6% of the assessments. The overall IRR was good, with a Kappa value of 0.78 (0.69–0.87, 95% CI) for final triage level. Comparison with reference group assessments established that over- and undertriage had occurred in 9% and 21% of the cases, respectively. The main reasons for undertriage were “not acknowledging abnormal vital sign parameters” and “limitations in study design”.ConclusionThe GOTS is a reliable tool for triaging obstetric patients. It enables a standardized triage process unrelated to the assessors’ level of experience in assessing and managing obstetric patients and is applicable for triaging obstetric patients presenting for emergency care at obstetric or emergency units.

Norwegian Clinicians’ Experiences of Learnability and Usability of SCID-II, SCID-5-PD and SCID-5-AMPD-I Interviews: A Sequential Multi-Group Qualitative Approach

The DSM-5 presents two competing diagnostic frameworks for personality disorders: the standard categorical model and the Alternative Model of Personality Disorders (AMPD). The AMPD was initially criticized for being too complex and theory laden for clinical implementation. Though inter-rater reliability studies have contested initial claims of the model’s complexity, little attention has been paid to how clinicians experience the usability and learnability of either model. We interviewed twenty Norwegian clinicians about their experiences with either the SCID-II/5-PD (n = 9), SCID-5-AMPD-I (n = 8), or both (n = 3). Separate thematic analyses were conducted for SCID-II/5-PD and SCID-5-AMPD-I groups, and group themes were compared. We identified four themes for each group, relating to required skills, training, challenges and information gained through the interview. We found that training and clinical experience were considered to be important for both interviews. Moreover, the SCID-5-AMPD-I was considered to rely more explicitly on theory specific to the development and content of the AMPD model in general and the LPFS specifically We also identified shared and unique challenges and shortcomings of each interview. We comment on how our findings relate to the debate surrounding the AMPD, and recommend development of clear training guidelines for both interviews.

Clinical assessment of pelvic floor and abdominal muscles 3 months post partum: an inter-rater reliability study

ObjectivesEvaluation of the inter-rater reliability of clinical assessment methods for pelvic floor muscles and diastasis recti abdominis post partum.DesignA multicentre inter-rater reliability study.SettingThree primary care rehabilitation centres in Sweden.ParticipantsA total...

THERapy-Related InterACTion (THER-I-ACT) in Rehabilitation-Instrument Development and Inter-Rater Reliability.

Objective: To develop an instrument for the observation of therapeutic communication interactions during rehabilitation sessions and test its inter-rater reliability.Methods: The new instrument THER-I-ACT (THERapy–related Inter-ACTion) has been designed to assess both the frequency and timing of therapeutic interactions in the thematic fields information provision, feedback, other motivational interaction, and bonding. For this inter-rater reliability study, a sample of stroke survivors received arm rehabilitation as either arm ability training, arm basis training, or mirror therapy, or neglect training as individually indicated. Therapy sessions were video-recorded (one for each participant) and therapeutic interactions rated by two independent raters using THER-I-ACT.Results: With regard to the instrument's comprehensiveness to document therapeutic interactions with pre-defined categories the data from 29 sessions suggested almost complete coverage. Inter-rater reliability was very high both for individual categories of therapeutic interaction (frequency and time used for interaction) (intraclass correlation coefficient, ICC 0.91–1.00) and summary scores for the thematic fields of interaction (again for frequency and time used for interaction) (ICC 0.98–1.00).The inter-rater reliability for rating engagement and being focussed for both the therapist and patient was substantial (ICC 0.71 and 0.86).Conclusions: The observational study documented that by use of the newly designed THER-I-ACT various types of therapy-related communication interactions performed by therapists can be assessed with a very high inter-rater reliability. In addition, the thematic fields and categories of therapeutic interaction as defined by the instrument comprehensively covered the type of interaction that occurred in the therapeutic sessions observed.

An interrater reliability study of gait analysis systems with the dual task paradigm in healthy young and older adults

Background and aimsOne reason for the controversial discussion of whether the dual task (DT) walking paradigm has an added value for diagnosis in clinical conditions might be the use of different gait measurement systems. Therefore, the purpose was 1) to detect DT effects of central gait parameters obtained from five different gait analysis devices in young and old adults, 2) to assess the consistency of the measurement systems, and 3) to determine if the absolut and proportional DT costs (DTC) are greater than the system-measurement error under ST.MethodsTwelve old (72.2 ± 7.9y) and 14 young adults (28.3 ± 6.2y) walked a 14.7-m distance under ST and DT at a self-selected gait velocity. Interrater reliability, precision of the measurement and sensitivity to change were calculated under ST and DT.ResultsAn age effect was observed in almost all gait parameters for the ST condition. For DT only differences for stride length (p < .029, ɳ2p = .239) as well as single and double limb support (p = .036, ɳ2p = .227; p = .034, ɳ2p = .218) remained. The measurement systems showed a lower absolute agreement compared to consistency across all systems.ConclusionsWhen reporting DT effects, the real changes in performance and random measurement errors should always be accounted for. These findings have strong implications for interpreting DT effects.

Workplace-Based Entrustment Scales for the Core EPAs: A Multisite Comparison of Validity Evidence for Two Proposed Instruments Using Structured Vignettes and Trained Raters.

In undergraduate medical education (UME), competency-based medical education has been operationalized through the 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs). Direct observation in the workplace using rigorous, valid, reliable measures is required to inform summative decisions about graduates' readiness for residency. The purpose of this study is to investigate the validity evidence of 2 proposed workplace-based entrustment scales. The authors of this multisite, randomized, experimental study used structured vignettes and experienced raters to examine validity evidence of the Ottawa scale and the UME supervisory tool (Chen scale) in 2019. The authors used a series of 8 cases (6 developed de novo) depicting learners at preentrustable (less-developed) and entrustable (more-developed) skill levels across 5 Core EPAs. Participants from Core EPA pilot institutions rated learner performance using either the Ottawa or Chen scale. The authors used descriptive statistics and analysis of variance to examine data trends and compare ratings, conducted interrater reliability and generalizability studies to evaluate consistency among participants, and performed a content analysis of narrative comments. Fifty clinician-educators from 10 institutions participated, yielding 579 discrete EPA assessments. Both Ottawa and Chen scales differentiated between less- and more-developed skill levels (P < .001). The interclass correlation was good to excellent for all EPAs using Ottawa (range, 0.68-0.91) and fair to excellent using Chen (range, 0.54-0.83). Generalizability analysis revealed substantial variance in ratings attributable to the learner-EPA interaction (59.6% for Ottawa; 48.9% for Chen) suggesting variability for ratings was appropriately associated with performance on individual EPAs. In a structured setting, both the Ottawa and Chen scales distinguished between preentrustable and entrustable learners; however, the Ottawa scale demonstrated more desirable characteristics. These findings represent a critical step forward in developing valid, reliable instruments to measure learner progression toward entrustment for the Core EPAs.

Dyspnea could be accurately assessed by a caregiver in hospitalized patients with respiratory diseases: Interrater reliability and agreement study

BackgroundThe perception of dyspnea is a subjective feeling typically self-assessed by the patient. However, the assessment by a caregiver is sometimes required. ObjectivesThe primary aim was to compare patient self-assessment and caregiver assessment of dyspnea (interrater reliability) using the modified Borg and visual analog scale (VAS) in hospitalized patients. The secondary aim was to compare dyspnea assessment between the two scales for patients and caregiver (inter-instrument reliability). MethodsSelf-assessment of dyspnea intensity of hospitalized patients with respiratory diseases was compared with caregiver's assessment. Dyspnea intensity was measured using two scales, the modified Borg scale (0–10 scale) and the 10 cm VAS. Mean difference and 95% confidence interval (CI) between assessors (i.e. patient versus caregiver) were calculated for each scale. Inter- and intra-rater reliability was calculated using intraclass correlation coefficients (ICCs). ResultsA total of 254 patients were recruited. The mean differences between patient and caregiver ratings were 0.31 (95% CI: 0.09, 0.53) for the modified Borg scale and 0.36 (95% CI: 0.06, 0.65) for the VAS scale. Interrater reliability was good for both scales with ICC of 0.79 (95% CI: 0.73, 0.84) for VAS and 0.82 (95% CI: 0.77, 0.86) for the modified Borg scale. The mean differences in scores between scales were 0.93 (95% CI 0.69, 1.17) for patients' ratings and 0.88 (95% CI 0.72, 1.04) for caregiver's rating. The inter-instrument reliability was moderate to good and similar for both assessors. ConclusionDyspnea can be accurately estimated by caregivers when patients with lung diseases cannot self-report. Scores on the VAS to rate dyspnea were higher than the scores on the Borg scale.

805 Towards a more perfect union: interrater reliability between telemedicine and in-person evaluators in sleep disordered breathing

Abstract Introduction Telemedicine has rapidly changed the landscape of all of clinical practice, and is now widely employed in sleep medicine. To date, the accuracy of telemedicine in identifying patients at risk for obstructive sleep apnea (OSA) is still unknown. Given differences in technique and fidelity, a perfect correlation between these two types of encounters cannot be assumed. We studied how providers using a telemedicine platform compared to providers using traditional in-person encounters in identifying risk for sleep disordered breathing. Methods 90 participants referred to a comprehensive university sleep program were randomized to this interrater reliability study. Subjects were representative of the gender and ethnic breakdown of the outlying community. The subjects were evaluated by an in-person clinician investigator, then randomized to a second clinician investigator who performed an evaluation online using a common teleconferencing platform. Both types of evaluations included a history and physical exam. The primary outcome was pretest probability for obstructive sleep apnea. Secondary comparators included level of daytime sleepiness, snoring volume, apneas witnessed by a third party, modified Mallampati score, presence/absence of tonsils, degree of overjet bite, and severity of apnea based on home sleep testing. Results Interrater reliability for pretest probability of OSA was reflected in a weighted kappa value of 0.414 (SE 0.090, p=0.002). This suggests moderate agreement between the an in-person and telemedicine evaluator. Kappa values of our secondary outcomes ranged from -0.044 (degree of maxillary overjet) to 0.702 (apneas witnessed by a third party), and were generally higher for historical elements and lower for physical exam findings. Conclusion Evaluation for pretest probability for sleep apnea via telemedicine has a moderate interrater correlation with in-person assessment. A relatively high degree of interrater reliability for historical elements suggests that the accuracy of telemedicine for OSA is tempered by a suboptimal physical exam. Telemedicine evaluations might become more accurate through standardization. For instance, using validated scales for OSA or templated encounter scripts may help with risk-stratification, and ultimately lead to more uniform management. Support (if any) This study was supported by an AASM Foundation Focused Project Award

Assessing the Quality of Mobile Health-Related Apps: Interrater Reliability Study of Two Guides.

BackgroundThere is a huge number of health-related apps available, and the numbers are growing fast. However, many of them have been developed without any kind of quality control. In an attempt to contribute to the development of high-quality apps and enable existing apps to be assessed, several guides have been developed.ObjectiveThe main aim of this study was to study the interrater reliability of a new guide — the Mobile App Development and Assessment Guide (MAG) — and compare it with one of the most used guides in the field, the Mobile App Rating Scale (MARS). Moreover, we also focused on whether the interrater reliability of the measures is consistent across multiple types of apps and stakeholders.MethodsIn order to study the interrater reliability of the MAG and MARS, we evaluated the 4 most downloaded health apps for chronic health conditions in the medical category of IOS and Android devices (ie, App Store and Google Play). A group of 8 reviewers, representative of individuals that would be most knowledgeable and interested in the use and development of health-related apps and including different types of stakeholders such as clinical researchers, engineers, health care professionals, and end users as potential patients, independently evaluated the quality of the apps using the MAG and MARS. We calculated the Krippendorff alpha for every category in the 2 guides, for each type of reviewer and every app, separately and combined, to study the interrater reliability.ResultsOnly a few categories of the MAG and MARS demonstrated a high interrater reliability. Although the MAG was found to be superior, there was considerable variation in the scores between the different types of reviewers. The categories with the highest interrater reliability in MAG were “Security” (α=0.78) and “Privacy” (α=0.73). In addition, 2 other categories, “Usability” and “Safety,” were very close to compliance (health care professionals: α=0.62 and 0.61, respectively). The total interrater reliability of the MAG (ie, for all categories) was 0.45, whereas the total interrater reliability of the MARS was 0.29.ConclusionsThis study shows that some categories of MAG have significant interrater reliability. Importantly, the data show that the MAG scores are better than the ones provided by the MARS, which is the most commonly used guide in the area. However, there is great variability in the responses, which seems to be associated with subjective interpretation by the reviewers.

Flexor digitorum superficialis muscular activity is more reliable than mentalis muscular activity for rapid eye movement sleep without atonia quantification: A study of interrater reliability for artifact correction in the context of semiautomated scoring of rapid eye movement sleep without atonia.

To evaluate interrater reliability for artifact correction in the context of semiautomated quantification of rapid eye movement (REM) sleep without atonia (RWA) in the mentalis and flexor digitorum superficialis (FDS) muscles. We included video-polysomnographies of 14 subjects with apnea-hypopnea index in REM sleep (AHIREM) < 15/h and 11 subjects with AHIREM ≥ 15/h. Eight subjects had isolated REM sleep behavior disorder. A validated algorithm (www.osg.be) automatically scored phasic and "any" EMG activity in the mentalis muscle, and phasic EMG activity in the FDS muscles. Four independent expert scorers performed artifact correction according to the SINBAR (Sleep Innsbruck Barcelona) recommendations. Interrater reliability for artifact correction was computed with B-statistics. The variability across scorers of four RWA indices (phasic mentalis, "any" mentalis, phasic FDS and SINBAR-i.e. "any" mentalis and/or phasic FDS-EMG activity indices) was computed. With Friedman tests, we compared B-statistics obtained for mentalis and FDS muscles, and the variability of the RWA indices. Influence of AHIREM and REM sleep behavior disorder (RBD) diagnosis on the RWA indices variability was evaluated with linear regressions. Interrater reliability for artifact correction was higher in the FDS than in the mentalis muscle (p < 0.001). Phasic FDS activity was minimally affected by artifacts. Accordingly, the phasic FDS EMG activity index had the lowest variability across scorers (p < 0.001). Variability across scorers of the RWA indices including the mentalis muscle increased with AHIREM and was independent from RBD diagnosis. Due to the consistently found low number of artifacts, phasic FDS activity is a reliable measure of RWA.

Rating criteria of the causal nexus in Whiplash Associated Disorders: A study of interrater reliability

IntroductionThe diagnosis of Whiplash Associated Disorders is subject to various criteria, some of a subjective nature, which impedes a high percentage of agreement among raters of the causal nexus. It is necessary to develop an evidence-based methodology to maximize the uniformity of this diagnosis. Our objective is to assess the interrater reliability of four experts in legal medicine who establish the causal nexus of traffic accident patients. Material and method300 traffic accident patients evaluated by 4 legal medicine experts. They conducted two ratings, one without and one with a study of the biomechanics of the accident. The degree of concordance of the ratings was studied by the intraclass correlation coefficient (ICC) and the concordance index (CI). ResultsThe results of both the intraclass correlation coefficient and the concordance index of the rating prior to the biomechanical study of the injury yield moderate-high values (ICC=.70; CI=.60). Likewise, the values observed in the rating with the study of biomechanics also indicate moderate-high agreement (ICC=.78, CI=.67). ConclusionsThe study of biomechanics slightly improves concordance. However, the aforementioned study does not help to reduce the proportion of doubtful cases and the levels of improvement are not acceptable. There are significant differences between the classifications, suggesting that the proposed valuation methodology is not uniform enough to reach a consensus.

Categorisation of lumbar spine MRI referrals in Denmark as compliant or non-compliant to international imaging guidelines: an inter-rater reliability study

BackgroundManaging low back pain (LBP) often involves MRI despite the fact that international guidelines do not recommend routine imaging. To allow us to explore the topic and use this knowledge in further research, a reliable method to review the MRI referrals is needed. Consequently, this study aimed to assess the inter-rater reliability of a method evaluating lumbar spine MRI referrals’ appropriateness.MethodsFour inexperienced students (chiropractic master’s students) and a senior clinician (chiropractor) were included as independent raters in this inter-rater reliability study. Lumbar spine MRI referrals from primary care on patients (> 18 years) with LBP with or without leg pain were included. The referrals were classified using a modified version of the American College of Radiology (ACR) imaging appropriateness criteria for LBP. Categories of appropriate referrals included; fractures, cancer, previous surgery, candidate for surgery or suspicion of cauda equina. Inappropriate referrals included lacking information on previous non-surgical treatment, no word on non-surgical treatment duration, or “other reasons” for inappropriate referrals.After two rounds of training and consensus sessions, 50 lumbar spine MRI referrals were reviewed independently by the five raters. Inter-rater reliability was quantified using unweighted Kappa statistics, and the observed agreement was calculated with both a pairwise comparison and an overall five-rater comparison.ResultsInter-rater reliability was substantial, with a Kappa value for appropriate vs. inappropriate referrals of 0.76 (95% CI: 0.55–0.89). When six and eight subcategories were evaluated, the Kappa values were 0.77 (95% CI: 0.58–0.91) and 0.82 (95% CI: 0.72–0.92), respectively.The overall percentage of agreement for appropriate and inappropriate referrals was 92% and ranged from 88 to 98% for the pairwise comparisons of the five raters’ results. For the six and eight subcategories, the overall agreement was 92 and 88%, respectively, ranging from 88 to 98% and 84–92%, respectively, for the pairwise comparisons.ConclusionThe inter-rater reliability of the evaluation of the appropriateness of lumbar spine MRI referrals, according to the modified ACR-appropriateness criteria, was found to range from substantial to almost perfect and can be used for research and quality assurance purposes.

Interrater reliability between in-person and telemedicine evaluations in obstructive sleep apnea.

We examined how telemedicine evaluation compares to face-to-face evaluation in identifying risk for sleep-disordered breathing. This was a randomized interrater reliability study of 90 participants referred to a university sleep center. Participants were evaluated by a clinician investigator seeing the patient in-person, then randomized to a second clinician investigator who performed a patient evaluation online via audio-video conferencing. The primary comparator was pretest probability for obstructive sleep apnea. The primary outcome comparing pretest probability for obstructive sleep apnea showed a weighted kappa value of 0.414 (standard error 0.090, P = .002), suggesting moderate agreement between the 2 raters. Kappa values of our secondary outcomes varied widely, but the kappa values were lower for physical exam findings compared to historical elements. Evaluation for pretest probability for obstructive sleep apnea via telemedicine has a moderate interrater correlation with in-person assessment. A low degree of interrater reliability for physical exam elements suggests telemedicine assessment for obstructive sleep apnea could be hampered by a suboptimal physical exam. Employing standardized scales for obstructive sleep apnea when performing telemedicine evaluations may help with risk-stratification and ultimately lead to more tailored clinical management.

Thumb carpometacarpal palmar and radial abduction in adults with thumb carpometacarpal joint pain: Inter-rater reliability and precision of the inter-metacarpal distance method

Study DesignPsychometric study of inter-rater reliability. IntroductionFunctional use of the thumb can be limited in individuals with thumb carpometacarpal (CMC) osteoarthritis(OA), especially in the presence of a thumb adduction contracture. Goniometry is a common method of assessing palmar and radial abduction of the thumb base and can be used as a method of determining effectiveness of an intervention for adduction contracture. However, goniometry for the assessment of these motions has been shown to have low to moderate reliability. The intermetacarpal distance (IMD) measurement method has been shown to be the most reliable for measuring CMC palmar abduction in individuals with healthy hands but has not been studied in persons with thumb CMC OA. PurposesThe purpose of this study was to determine the inter-rater reliability and precision of the inter-metacarpal distance method for measuring palmar and radial abduction in persons with symptoms of thumb CMC OA. MethodsTwo trained hand therapists utilized the IMD method to measure palmar and radial abduction in the affected hands of 22 subjects (28 thumbs) with a physician-confirmed diagnosis or positive provocative test consistent with a diagnosis of thumb CMC OA. The intraclass correlation coefficient (ICC2,2) was used to assess inter-rater reliability of the IMD method. To determine the precision of the measurements, the standard error of measurement (SEM), minimal detectable change (MDC), and MDC percent were calculated. Findings were supplemented with descriptive data on the IMD values as well as descriptive data on the sample. ResultsIntraclass correlation coefficients for both radial and palmar abduction were found to be >.75, indicating excellent reliability. The precision of the IMD measurements were acceptable-to-excellent as evidenced by MDC% values of <30% and <10% for radial and palmar abduction respectively. ConclusionsWe present a new method for measuring thumb radial abduction. The inter-metacarpal distance method has excellent inter-rater reliability and acceptable-to-excellent precision when measuring palmar and radial abduction in individuals with or suspected to have thumb CMC OA. Currently, it is the most reliable tool for measuring thumb abduction.