Poor Intrarater Reliability Research Articles

Inter-rater and intrarater reliability of superior labrum anterior to posterior lesion classification using magnetic resonance arthrography

BackgroundThe glenoid labrum is a fibrocartilaginous ring that affixes the joint capsule and ligaments of the glenohumeral joint2. Superior labrum anterior to posterior (SLAP) lesions are a subset of injuries that affect the superior glenoid labrum, most common in laborers and overhead-throwing athletes11. In 1990, Snyder et al classified SLAP lesions into one of four types10. Later, Maffet et al expanded this scale to include three additional subclassifications8. At present, arthroscopy is considered the gold standard for SLAP tear diagnosis5. Classification under arthroscopy has demonstrated low to moderate inter-rater reliability3. MRa is an alternate, less invasive test for diagnosing SLAP lesions. The reliability of MRa for diagnosing slap tears is uncertain. MethodsMR arthrograms were identified using the Picture Archiving and Communication System (PACS). In total, 273 shoulder arthrograms were reviewed, and 20 were selected with the desired pathology. Three orthopedic surgeons and three musculoskeletal radiologists were asked to classify the SLAP lesions into one of seven categories (Snyder & Maffet classification systems). Data was collected on two separate occasions at an interval of at least two months. Inter- and intra-rater reliability were calculated using Fleiss Kappa and Cohen's Kappa, respectively. ResultsBetween all raters, there was poor inter-rater reliability for each round of data collection (κ = .177, κ = .124 for rounds 1 and 2, respectively). Between orthopedic surgeons, there were poor levels of agreement (κ = -.056, κ = .114), whereas, between radiologists, there was fair to moderate agreement (κ = 0.479, κ = 0.340). Within orthopedic raters, κ values ranged from -0.059 to 0.125, indicating, at best, poor intra-rater reliability. Within radiologists, κ values ranged from 0.545 to 0.553, indicating moderate agreement within raters. The analysis determined that none of the orthopedic values for inter or intra-rater reliability could be deemed statistically different from zero. ConclusionOverall, classification using MRa resulted in significant disagreement between and within raters. Trained radiologists demonstrated higher overall levels of agreement than orthopedic surgeons. In summary, when using MRa to assess SLAP lesions, Snyder and Maffet classification demonstrates poor reliability by orthopedic surgeons and moderate reliability when used by musculoskeletal radiologists.

Impact of operator experience on transcranial magnetic stimulation

ObjectiveTo determine the impact of an operator’s experience on transcranial magnetic stimulation (TMS) measurement. MethodsOperator B (beginner), operator E (expert), and 30 healthy participants joined the study consisting of two experiments. In each experiment, each operator performed a TMS protocol on each participant in a random order. ResultsCompared with operator E, operator B exhibited higher resting motor threshold (RMT) in experiment I (60.1 ± 13.0 vs. 57.4 ± 10.9% maximal stimulation output, p = 0.017) and the difference disappeared in experiment II (p = 0.816). In 1-mV motor evoked potential (MEP) measurement, operator B exhibited higher standard deviation indicating lower consistency in experiment I compared with experiment II (1.05 ± 0.40 vs. 1.05 ± 0.16 mV with unequal variances, p = 0.001) and had poor intrarater reliability between the experiments (intraclass correlation coefficient = −0.130). There was no difference in the results of active motor threshold, silent period, paired-pulse stimulation, or continuous theta burst stimulation between the operators. ConclusionsAn operator’s experience in TMS may affect the results of RMT measurement. With practice, a beginner may choose a more precise stimulation location and have higher consistency in 1-mV MEP measurement. SignificanceWe recommend that a beginner needs to practice for precise stimulation locations before conducting a trial or clinical practice.

Evaluation of the International Spinal Cord Injury Bowel Function Basic Data Set Version 2.0 in Children and Youth With Spinal Cord Injury.

To establish the utility and feasibility of the International Spinal Cord Injury/Dysfunction (SCI/D) Bowel Function Basic Data Set Version 2.0 in pediatric SCI populations. This was a noninterventional, repeated measure design conducted in Pennsylvania, Maryland, Illinois, Kentucky, and South Carolina. The International Spinal Cord Injury/Dysfunction (SCI/D) Bowel Function Basic Data Set Version 2.0 was administered repeatedly, twice at the point of care and once over the phone. Time to complete the data set was recorded. Inter- and intrarater reliability was examined by intraclass correlation coefficients (ICC) with 95% confidence intervals (CI), and agreement between the bowel function basic data set variables and medical records was calculated using percentages. Intrarater reliability involved the same person administering the data once at the point of care and once over the phone. Forty-one children/youth ages 1 to 20 years participated in this study. Average time to complete the data set was 5.17 minutes. Interrater reliability was good to excellent (ICC ≥ 0.75) for most variables. Five variables had moderate interrater reliability (ICC = 0.05-0.74) and three had poor interrater reliability (ICC < 0.05). With the exception of one variable that had poor intrarater reliability (constipating agent, ICC = 0.00) and one that approached moderate reliability (digital evacuation, ICC = 0.74), intrarater reliability was good to strong for every bowel variable (ICC = 0.88-1.00). Only 12 (32%) medical records had explicit documentation of one or more of the variables on the Basic Bowel Function Basic Data Set V2.0. The results support future research with a larger and more diverse sample of children with SCI to build upon the psychometric work described herein.

Technical report: Inter- and intra-rater reliability of regional gastrointestinal transit times measured using the 3D-Transit electromagnet tracking system.

The 3D-Transit electromagnet tracking system is an emerging tool for the ambulatory assessment of gastrointestinal (GI) transit times and motility patterns, based on the anatomical localization of ingestible electromagnetic capsules. Currently, 3D-Transit recordings are manually analyzed to extract GI transit times. As this is a subjective method, there is some inherent variability in the measurements, which may be experience-dependent. We therefore assessed inter- and intra-rater reliability of GI transit times from 3D-Transit recordings. Thirty-six 3D-Transit recordings (17 female; median age: 34years [range: 21-80]) were analyzed twice by 3 raters with varying experience. Each rater manually identified the timestamps when a capsule progressed from antrum to duodenum, and from ileum to right colon. These timestamps, along with the ingestion and expulsion times, were used to determine whole gut (WGTT), gastric emptying (GET), small intestinal (SITT) and colonic (CTT) transit times. Reliability was determined using interclass correlation coefficients (ICCs). For capsule progression timestamps, the most and mid-experienced raters had fair to good inter- and excellent intra-rater reliability (ICCmin-max =0.61-1.00), whereas the inexperienced rater had poor to fair inter- and poor intra-rater reliability (ICCmin-max =0.28-0.55). GET and SITT reliability between the most and mid-experienced raters was fair (ICCmin-max =0.61-0.73), while reliability between these raters and the inexperienced rater was poor to fair (ICCmin-max =0.28-0.55). CTT reliability was excellent between and within all raters (ICCmin-max =0.92-0.99). Inexperienced raters provide the least reliable measurements from 3D-Transit recordings, which confirms requirement for adequate training. Automation may improve the reliability of measurements.

Validation of Adapted Dartmouth Operative Conditions Scale for sedation during pediatric esophagogastroduodenoscopy.

Pediatric esophagogastroduodenoscopy requires deep sedation as it involves stimulation of the airway. Frequency of adverse events is higher with esophagogastroduodenoscopy. Hence, monitoring needs sedation scales like the Dartmouth Operative Condition Scale that identifies safe states of sedation. This study aims at validating the Adapted Dartmouth Operative Condition Scale for sedation rating by pediatricians during pediatric esophagogastroduodenoscopy. Items in the Dartmouth Operating Conditions Scale were adapted for pediatric esophagogastroduodenoscopy. Videos of 35 procedures were recorded. The recording started 10 min before sedation and continued till recovery. The videos were split into preprocedure videos, intraprocedure videos, and recovery videos, and further split into 30-s clips. Twenty representative intraprocedure videos clips were selected. Ten raters scored the videos with the Adapted Dartmouth Operating Conditions Scale and modified-COMFORT score. The Adapted Dartmouth Operating Conditions Scale scoring was repeated after a month. The preprocedure videos, intraprocedure videos, and recovery videos of ten patients (six clips each) were scored by one rater to assess responsiveness. The Adapted Dartmouth Operating Conditions Scale detected nonoptimal sedation states including laryngospasm due to undersedated states. It showed fair interrater reliability at timeline-1 (intraclass correlation coefficient = 0.45) and timeline-2 (intraclass correlation coefficient = 0.65) but poor intrarater reliability (intraclass correlation coefficient = 0.32). There was significant positive correlation with modified-COMFORT (Spearman's rank order correlation, r = 0.150). Responsiveness was demonstrated by the difference in the preprocedure videos, intraprocedure videos, and recovery videos scores (F = 126.50). The Adapted Dartmouth Operating Conditions Scale detects nonoptimal sedation states during pediatric-esophagogastroduodenoscopy. It shows good criterion validity, interrater reliability, and responsiveness. Poor intrarater reliability seen in our study could be due to item ambiguity arising from the mode of training of the raters in the Dartmouth Operating Conditions Scale.

Establishing Inter- and Intrarater Reliability for High-Stakes Testing Using Simulation.

This article reports one method to develop a standardized training method to establish the inter- and intrarater reliability of a group of raters for high-stakes testing. Simulation is used increasingly for high-stakes testing, but without research into the development of inter- and intrarater reliability for raters. Eleven raters were trained using a standardized methodology. Raters scored 28 student videos over a six-week period. Raters then rescored all videos over a two-day period to establish both intra- and interrater reliability. One rater demonstrated poor intrarater reliability; a second rater failed all students. Kappa statistics improved from the moderate to substantial agreement range with the exclusion of the two outlier raters' scores. There may be faculty who, for different reasons, should not be included in high-stakes testing evaluations. All faculty are content experts, but not all are expert evaluators.

Audio-Perceptual Evaluation of Portuguese Voice Disorders—An Inter- and Intrajudge Reliability Study

The aim of this article was to describe the results of an audio-perceptual evaluation carried out by 10 judges, on a database comprising 90 voice recordings plus 10 samples repetition, with the purpose of characterizing the intra- and interrater reliability. Exploratory, transversal. The classification of the GRBAS parameters was obtained for each one of the 10 experts, concerning the 90 voice samples. The intraclass correlation coefficient determined the interrater reliability. For the 10 repeated voices, the intrarater reliability was assessed by means of a dispersion analysis. The average judges' classification for each of the GRBAS parameters differs (P<0.05). The values of the correlations, with confidence intervals of 95%, between the average scores for all components of the GRBAS scale lie, in general, between 0.838 and 0.966. The first three parameters of the scale (G, R, and B) have the higher interrater reliability. Differences were statistically significant (P<0.05) for experts 1, 6, 9, and 10, which means a poor intrarater reliability for 40% of the judges. All the experts had similar evaluation criteria for the assessment of the five parameters of the GRBAS scale (the values of the confidence intervals at 95% of the experts average ratings of the GRB were above 0.8). However, its quantification is not statistically similar. Asthenia and Strain have lower reliability. Most experts do not reveal statistically significant differences between the values assigned to the GRB parameters (P>0.05).

Introducing a new definition of a near fall: Intra-rater and inter-rater reliability

Near falls (NFs) are more frequent than falls, and may occur before falls, potentially predicting fall risk. As such, identification of a NF is important. We aimed to assess intra and inter-rater reliability of the traditional definition of a NF and to demonstrate the potential utility of a new definition. To this end, 10 older adults, 10 idiopathic elderly fallers, and 10 patients with Parkinson's disease (PD) walked in an obstacle course while wearing a safety harness. All walks were videotaped. Forty-nine video segments were extracted to create 2 clips each of 8.48min. Four raters scored each event using the traditional definition and, two weeks later, using the new definition. A fifth rater used only the new definition. Intra-rater reliability was determined using Kappa (K) statistics and inter-rater reliability was determined using ICC. Using the traditional definition, three raters had poor intra-rater reliability (K<0.054, p>0.137) and one rater had moderate intra-rater reliability (K=0.624, p<0.001). With the traditional definition, inter-rater reliability between the four raters was moderate (ICC=0.667, p<0.001). In contrast, the new NF definition showed high intra-rater (K>0.601, p<0.001) and excellent inter-rater reliability (ICC=0.815, p<0.001). A priori, it is easy to distinguish falls from usual walking and NFs, but it is more challenging to distinguish NFs from obstacle negotiation and usual walking. Therefore, a more precise definition of NF is required. The results of the present study suggest that the proposed new definition increases intra and inter-rater reliability, a critical step for using NFs to quantify fall risk.

Development of a tool to guide referral of patients with neuromuscular disorders to allied health services. Part two

Purpose. The Perceived Limitations and Needs Questionnaire (PLAN-Q) was developed to guide referral for a one-off consultation by occupational therapy (OT), physical therapy (PT), and speech therapy (ST) consultation, to provide a tailor-made advice on allied health interventions. This article reports on the testing of validity and reliability of the PLAN-Q.Methods. In the validation study, 208 patients with a broad spectrum of neuromuscular disorders completed the PLAN-Q, Medical Outcome Study short-form 36-item version (SF-36), and the Impact of Participation on Autonomy (IPA) questionnaires. A subsection of 51 patients, whose physical condition was stable, participated in the evaluation of the intra-rater reliability of the questionnaire. The theoretical construct was tested with factorial analysis, subscales were constructed and reliability and validity of the PLAN-Q subscales were assessed.Results. Factorial analysis resulted in an 18-item self-report questionnaire. Items were grouped into four subscales (‘physical capacity’, ‘transferring’, ‘oropharyngeal capacity’, and ‘hand-use’), each with two dimensions (‘capacity’ and ‘need for help’). The internal consistency of all subscales was good (Cronbach's α: 0.77 – 0.94) as well as the intra-rater reliability of the subscales ‘physical capacity’ and ‘transferring’ in the ‘capacity’ dimension (Kappa: 0.70 – 0.75). The ‘need’ dimension showed poor intra-rater reliability suggesting that ‘need for help’ is a variable phenomenon that changes between two points of measurement. The construct validity of the subscales against the SF-36 and the IPA was satisfactory.Conclusion. The PLAN-Q is a valid self-report instrument that measures patients' perception of capacity and needs in domains relevant to referral for a one-off OT, PT, or ST consultation.