Assess Interrater Reliability Research Articles

Intra- and Interrater Reliability of Remote Assessment of Transfers by Wheelchair Users Using the Transfer Assessment Instrument (Version 4.0)

ObjectiveTo evaluate the reliability, standard error of measurement, minimum detectable change, and item-level consistency of the Transfer Assessment Instrument (TAI) version 4.0 when used to evaluate transfer quality remotely. DesignParticipants transferred from their wheelchair to a mat table (transfer 1), repeated this after a 10-minute delay to assess intrarater reliability (transfer 2), and repeated this 1-2 days later to assess test-retest reliability (transfer 3). Each transfer was scored in person by 4 raters and asynchronously by a remote clinician rater. Setting2017 National Veterans Wheelchair Games. ParticipantsConvenience sample of 44 full-time wheelchair users (N=44). InterventionsNot applicable. Main Outcome MeasuresTAI total score, subscores (wheelchair setup, body setup, flight/landing), and item scores (15 items). ResultsModerate to excellent reliability was found when scoring remotely for TAI total and subscores for intrarater (intraclass correlation coefficient (ICC(3,1)=0.687-0.854), test-retest (ICC(3,1)=0.695-0.836), and interrater reliability (ICC(3,5)=0.746-0.962). Remote rater total score and flight/landing subscore were greater (indicating higher transfer quality) compared to the average in-person raters (P=.021 and P=.005, respectively). There were no differences between transfers 1-3 in remote rater scores. Item-level percentage agreement between the remote rater and in-person exceeded the 75% cutoff for clinical utility for all items. ConclusionsThe TAI is a reliable outcome measure for assessing transfer technique remotely.

The interrater reliability and agreement of a 0 to 10 uterine tone score in cesarean delivery

Postpartum hemorrhage is a leading source of maternal morbidity and mortality worldwide with uterine atony identified as the underlying cause in up to 80% of cases. Several measures have been utilized to report uterine tone. The most commonly reported measure is a 0 to 10 numeric rating scale, but this scale has not been tested for reliability or agreement between different raters. The primary purpose of this study was to evaluate the interrater reliability and agreement of the 0 to 10 visual numeric rating scale of uterine tone during cesarean delivery. A secondary purpose was to obtain estimates of scale responsiveness and minimal clinically important difference. Between August and November of 2018, obstetricians used a 0 to 10 numeric rating score to independently rate uterine tone at 3 and 10 minutes after cesarean delivery by palpation of the uterus. Of note, "0" represented "no tone" and "10" represented excellent tone. Each obstetrician independently and blinded to the other's score pointed to a numeric rating scale held by the anesthesiologist through a clear sterile drape. Intraclass correlation coefficients and Bland-Altman analysis were used to assess interrater reliability and agreement, respectively. Standardized response mean and standard error of measurement were used to obtain estimates of responsiveness and minimal clinically important difference, respectively. A total of 82 and 84 pairs of scores were collected at 3 and 10 minutes, respectively, from pairs of 62 unique obstetricians. The mean±standard deviation difference in scores between rater 1 and rater 2 was 0.4±1.4 at 3 minutes and 0.1±1.1 at 10 minutes. Intraclass correlation coefficients for a future single rater (intraclass correlation coefficient [1, 1]) at 3 and 10 minutes were 0.67 (95% confidence interval, 0.53-0.77) and 0.61 (95% confidence interval, 0.46-0.73), and for the average between 2 future raters (intraclass correlation coefficient [1, 2]), they were 0.80 (95% confidence interval, 0.71-0.87) and 0.76 (95% confidence interval, 0.63-0.84), indicating good and excellent reliability, respectively. Bland-Altman analysis estimated 95% limit of agreement between raters of -2.4 (95% confidence interval, -3.0 to -1.9) to 3.1 (95% confidence interval, 2.6-3.7) at 3 minutes and -2.1 (95% confidence interval, -2.5 to -1.7) to 2.4 (95% confidence interval, 2.0-2.8) at 10 minutes, consistent with good interrater agreement at both time points. The standardized response mean from 3 to 10 minutes after delivery was 1.1 (n=81). Standard error of measurement was 1.0 (95% confidence interval, 0.9-1.1) at 3 minutes and 0.8 (95% confidence interval, 0.7-0.9) at 10 minutes. The 0 to 10 numeric rating scale for uterine tone demonstrated good to excellent interrater reliability with 1 and 2 raters, respectively, and good interrater agreement. The scale was responsive to within-parturient change in tone, and preliminary estimates of the minimal clinically important difference were obtained. The 0 to 10 numeric rating scale for uterine tone may be a reliable, standardized tool for future research in reporting degree of uterotonic contraction during cesarean delivery.

Serial Neurologic Assessment in Pediatrics (SNAP): A New Tool for Bedside Neurologic Assessment of Critically Ill Children.

We developed a tool, Serial Neurologic Assessment in Pediatrics, to screen for neurologic changes in patients, including those who are intubated, are sedated, and/or have developmental disabilities. Our aims were to: 1) determine protocol adherence when performing Serial Neurologic Assessment in Pediatrics, 2) determine the interrater reliability between nurses, and 3) assess the feasibility and acceptability of using Serial Neurologic Assessment in Pediatrics compared with the Glasgow Coma Scale. Mixed-methods, observational cohort. Pediatric and neonatal ICUs. Critical care nurses and patients. None. Serial Neurologic Assessment in Pediatrics assesses Mental Status, Cranial Nerves, Communication, and Motor Function, with scales for children less than 6 months, greater than or equal to 6 months to less than 2 years, and greater than or equal to 2 years old. We assessed protocol adherence with standardized observations. We assessed the interrater reliability of independent Serial Neurologic Assessment in Pediatrics assessments between pairs of trained nurses by percent- and bias- adjusted kappa and percent agreement. Semistructured interviews with nurses evaluated acceptability and feasibility after nurses used Serial Neurologic Assessment in Pediatrics concurrently with Glasgow Coma Scale during routine care. Ninety-eight percent of nurses (43/44) had 100% protocol adherence on the standardized checklist. Forty-three nurses performed 387 paired Serial Neurologic Assessment in Pediatrics assessments (149 < 6 mo; 91 ≥ 6 mo to < 2 yr, and 147 ≥ 2 yr) on 299 patients. Interrater reliability was substantial to near-perfect across all components for each age-based Serial Neurologic Assessment in Pediatrics scale. Percent agreement was independent of developmental disabilities for all Serial Neurologic Assessment in Pediatrics components except Mental Status and lower extremity Motor Function for patients deemed "Able to Participate" with the assessment. Nurses reported that they felt Serial Neurologic Assessment in Pediatrics, compared with Glasgow Coma Scale, was easier to use and clearer in describing the neurologic status of patients who were intubated, were sedated, and/or had developmental disabilities. About 92% of nurses preferred to use Serial Neurologic Assessment in Pediatrics over Glasgow Coma Scale. When used by critical care nurses, Serial Neurologic Assessment in Pediatrics has excellent protocol adherence, substantial to near-perfect interrater reliability, and is feasible to implement. Further work will determine the sensitivity and specificity for detecting clinically meaningful neurologic decline.

Interrater Reliability among Novice Raters in the Assessment of Pelvic Floor Muscle Tone Using the Reissing Tone Scale.

Purpose: The authors sought to determine the interrater reliability among novice raters of intra-vaginal manual assessment of pubococcygeus muscle tone in women using the Reissing tone scale (RTS). Method: Three graduating physiotherapy students (novice raters) and one experienced pelvic floor physiotherapist assessed 31 female participants (aged 20-66 y). Assessors gave RTS scores for pubococcygeus tone at three intra-vaginal locations (6:00, 9:00, and 3:00). Interrater reliability was determined for the novice raters using a two-way random single-measures absolute agreement intra-class correlation coefficient (ICC). Spearman rank correlation (SRC) analysis determined the correlation between the novice and expert scores. Results: The ICC values for the novice raters were 0.523, 0.274, and 0.336 at 6:00, 9:00, and 3:00, and the SRC values between the novice and expert raters were 0.580, 0.320, and 0.340. Conclusions: The novice raters demonstrated low to moderate interrater reliability for intra-vaginal manual assessment of pubococcygeus tone. This result indicates that manual assessment of pelvic floor muscle tone is not reliable enough to use as a stand-alone test to guide treatment, at least for physiotherapists with limited clinical experience.

Clinical Audit on Examination Tools Used by Physical Therapists in Metro Manila in Examining Conditions with Neck Pain

Background: Neck pain is considered the fourth leading cause of disability, with an annual prevalence rate of 15 to 30%. Using evidence-based practice in neck pain examination is a vital part of the rehabilitation process as it serves as a basis for determining the best treatment. The objective of the study is to determine the usage of recommended examination tool for neck pain among the physical therapists in selected hospitals and clinics in Metro Manila. Methods: The study has three distinct phases wherein phase 1 was the development and validation of a data extraction sheet, phase 2 was the assessment of interrater reliability among the investigators who will perform the chart review, and phase 3 was the chart review process. Descriptive statistics were used for data analysis. Results: In phase 1, the contents of the data extraction sheet were found to be valid. In phase 2, the inter-rater reliability was 96.7% percent. In phase 3, the visual analogue scale was the most commonly used examination tool, yielding a 54% usage. This was followed by cervical range of motion &amp; cervical manual muscle testing (22%), palpation (15%), sensory testing (7%), postural assessment (6%), special test (4%), ocular inspection (2%), functional assessment (1%), Functional Index Measure (1%) and functional muscle testing (1%). Neck Disability Index, which was one of the literature-recommended examination tools, was not used. Conclusion: Visual analogue scale was the most commonly used examination tool in conditions with neck pain in selected hospitals and clinics in Metro Manila. Further investigation can be done in order to know the reasons for the use or nonuse of examination tools.

Taking "the boss" into the real world: Field interrater reliability of the Short-Term Assessment of Risk and Treatability: Adolescent Version.

There is emerging evidence that the performance of risk assessment instruments is weaker when used for clinical decision‐making than for research purposes. For instance, research has found lower agreement between evaluators when the risk assessments are conducted during routine practice. We examined the field interrater reliability of the Short‐Term Assessment of Risk and Treatability: Adolescent Version (START:AV). Clinicians in a Dutch secure youth care facility completed START:AV assessments as part of the treatment routine. Consistent with previous literature, interrater reliability of the items and total scores was lower than previously reported in non‐field studies. Nevertheless, moderate to good interrater reliability was found for final risk judgments on most adverse outcomes. Field studies provide insights into the actual performance of structured risk assessment in real‐world settings, exposing factors that affect reliability. This information is relevant for those who wish to implement structured risk assessment with a level of reliability that is defensible considering the high stakes.

Reliability of Speech-Language Pathologists' Categorizations of Preschoolers' Communication Impairments in Practice.

Purpose An efficient and reliable way to categorize children's communication impairments based on routine clinical assessments is needed to inform research and clinical decisions. This preliminary study assessed interrater reliability of speech-language pathologists' categorization of preschoolers' speech, language, and communication impairments using a clinical consensus document. Method Six speech-language pathologists at three community sites worked in pairs to assess 38 children aged 1-5 years, then used the clinical consensus document to categorize children's communication impairments broadly. Identified language and speech sound impairments were further subcategorized. Results Speech-language pathologists had substantial to almost perfect agreement for three broadly focused impairment categories. Agreement for whether language difficulties/disorders were developmental or associated with a biomedical condition was almost perfect, but moderate for whether difficulties impacted receptive or expressive language, or social communication skills. Agreement was fair for rule-based speech delays/disorders, but low for motor-based and mixed speech impairments. Conclusions Results support use of the clinical consensus document to collect data for reliable categories. Additional work is needed to confirm reliability for some broadly focused impairment categories and for subcategorization of speech impairments.

The General Movements Assessment in Neonates With Hypoxic Ischemic Encephalopathy.

Clinical measures after birth and studies such as electroencephalogram (EEG) and brain imaging do not fully predict neurodevelopmental outcomes of infants with hypoxic-ischemic encephalopathy. Early detection of adverse neurologic outcomes, and cerebral palsy in particular, in high-risk infants is essential for ensuring timely management. The General Movements Assessment is a tool that can be used in the early detection of cerebral palsy in infants with brain injury. The majority of studies on the General Movements Assessment in the late preterm and term population were performed prior to the introduction of therapeutic hypothermia. To apply the General Movements Assessment in late preterm and term infants with hypoxic-ischemic encephalopathy (including those who received therapeutic hypothermia), to determine if clinical markers of hypoxic-ischemic encephalopathy predict abnormal General Movements Assessment findings, and to evaluate interrater reliability of the General Movements Assessment in this population. Study design: Pilot prospective cohort study Subjects: We assessed 29 late preterm and full-term infants with mild, moderate, and severe hypoxic-ischemic encephalopathy in Philadelphia, PA. Most infants' general movements normalized by the fidgety age. Only infants with moderate or severe hypoxic-ischemic encephalopathy had abnormal general movements in both the writhing and the fidgety ages (n = 6). Seizure at any point during the initial hospitalization was the clinical sign most predictive of abnormal general movements in the fidgety age (sensitivity 100%, specificity 55%, positive predictive value 40%, negative predictive value 100%). Interrater reliability was greatest during the fidgety age (κ = 0.67). Seizures were the clinical predictor most closely associated with abnormal findings on the General Movements Assessment. However, clinical markers of hypoxic-ischemic encephalopathy are not fully predictive of abnormal General Movements Assessment findings. Larger future studies are needed to evaluate the associations between the General Movements Assessment and childhood neurologic outcomes in patients with hypoxic-ischemic encephalopathy who received therapeutic hypothermia.

PET/CT Poorly Predicts AJCC 8th Edition Pathologic Staging in HPV-Related Oropharyngeal Cancer.

The American Joint Committee on Cancer (AJCC) 8th edition introduced distinct clinical and pathological staging paradigms for human papilloma virus positive (HPV+) oropharyngeal squamous cell carcinoma (OPSCC). Treatment planning for OPSCC often utilizes positron emission tomography/computed tomography (PET/CT) to assess clinical stage. We hypothesize that PET/CT will accurately predict final pathologic AJCC 8th edition staging in patients with HPV+ OPSCC. All patients with primary HPV+ OPSCC with preoperative PET/CT who underwent transoral robotic surgery and neck dissection between 2011 and 2017 were identified. Data were collected via chart review. Two neuroradiologists performed blinded re-evaluation of all scans. Primary tumor size and cervical nodal disease characteristics were recorded and TNM staging was extrapolated. Cohen's kappa statistic was used to assess interrater reliability. Test for symmetry was performed to analyze discordance between radiologic and pathologic staging. Forty-nine patients met inclusion criteria. Interrater reliability was substantial between radiologists for nodal (N) and overall staging (OS) (κ=0.715 and 0.715). Radiologist A review resulted in identical OS for 67% of patients, overstaging for 31%, and understaging for 2%. Radiologist B review resulted in 61% identical OS, 39% overstaging, and 0% understaging. In misclassified cases, the test of symmetry shows strong bias toward overstaging N stage and OS (P < .001). Radiologic interpretation of extracapsular extension showed poor interrater reliability (κ=0.403) and poor accuracy. PET/CT predicts a higher nodal and overall stage than pathologic staging. PET/CT should not be relied upon for initial tumor staging, as increased FDG uptake is not specific for nodal metastases. PET/CT is shown to be a poor predictor of ECE. 4 Laryngoscope, 131:1535-1541, 2021.

Femoral Neck Shaft Angle and Management of Proximal Femur Fractures: Is the Contralateral Femur a Reliable Template?

To (1) assess interrater reliability of a novel technique for measurement of neck shaft angle (NSA); (2) use pelvic anteroposterior (AP) radiographs of unaffected hips to assess variability of NSA; and (3) evaluate the side-to-side variability of NSA to determine reliability of using the contralateral hip as a template. Retrospective cohort study. Academic Level 1 regional trauma center. Four hundred six femora (203 patients) with standing AP pelvis radiographs were selected. Exclusions included lack of acceptable imaging, congenital abnormalities, or prior hip surgery. An AP pelvis radiograph in the standing position. Bilateral NSA measurements obtained in a blinded fashion between 2 reviewers. Pearson coefficients and coefficient of determination assessed correlations and variability between left and right NSA. Concordance correlation coefficients assessed the interrater reliability between measurements performed by the 2 reviewers. Two hundred three patients (406 femora) were assessed. Male patients had a lower overall NSA mean of 131.56 degrees ± 4.74 than females with 133.61 degrees ± 5.17. There was no significant difference in NSA side-to-side in females (P = 0.18), 0.3 degrees [95% confidence interval (-0.15 to 0.75)], or males (P = 0.68), 0.19 degrees [95% confidence interval (-0.74 to 1.12)]. There was a strong linear relationship between left and right femora (r2 = 0.70). Forty-one percent of patients fell within the 131-135 degrees range bilaterally. Eighty-eight percent of patients had <5 degrees difference in NSA bilaterally and 0% had >10 degrees difference. There is no significant variability between bilateral femora in males and females. Use of this measurement method and contralateral NSA for proximal femur fracture planning is supported.

Assessment of inter-rater and intra-rater reliability of index of orthodontic treatment need index by newly trained orthodontic residents

Introduction: This study aimed to test the reliability of the index of treatment need by newly trained orthodontic residents. Materials and Methods: A total of 100 patients were assessed by the panel of four consultant orthodontists using the index of orthodontic treatment need (IOTN) index, and the agreed category was set as the 'gold standard'. Four residents of the department in the initial stages of formal orthodontic training and lacking the knowledge of using the IOTN index were trained to score the treatment need according to the IOTN index. All four residents were then asked to score the 100 sets of models using both the indexes of IOTN. Results: Kappa scores for inter-rater agreement with the expert panel for the major categories (1–4) demonstrated moderate agreement (kappa: 0.59–0.5623) for all raters. The percentage agreement ranged from 84.03% to 88.10% in all cases. Intra-rater agreement for the major categories was fair to moderate (kappa: 0.53–0.80).Conclusions: The IOTN index is a reliable index to be used for determining the priority of treatment needs in orthodontic treatment. The clarity and simplicity make the index easy to learn and apply with minimal training. The simplicity of the IOTN index makes it easy to learn and apply by novices.

Greek Translation, Content Validity, Test-Retest and Inter-Rater Reliability of Swimming with Independent Measure (SWIM) Assessment Test for People with Disabilities Based on Halliwick Concept

Objectives: Halliwick-based hydrotherapy has become one of the most widespread approaches in the water environment for people with disabilities. The Swimming with Independent Measure (SWIM) test was developed by Peackok (1993) based on the need to create specific aquatic assessment tests for people with disabilities. The purpose of our study was its translation into Greek language and the investigation of its metric properties. Methods: A total of 40 children with cerebral palsy aged on average 7.36 years participated in the study: 20 children with GMFCS 1 - 5 and cognitive limitations, and 20 children with GMFCS 1 - 4 without cognitive limitations. Two physiotherapists with extensive experience in hydrotherapy and Halliwick philosophy participated in the test-retest and inter-rater reliability and validity assessment. Results: The test-retest and inter-rater reliability for the overall score was found to be high (ICC = 0.99). In terms of validity, a positive correlation was found between SWIM and GMFM (r = 0.59). Conclusions: The Greek translation of the SWIM test was found to be highly reliable for assessing mental adaptation and functionality in the aquatic environment for children with disabilities, so it is recommended to professionals.

Reliability of Hand-Held Dynamometer for assessing Isometric Lumbar Muscles Strength in Asymptomatic Healthy Population.

Objective:To determine intra-rater and inter-rater reliability of hand-held dynamometer for assessing isometric lumbar muscle strength in asymptomatic healthy population.Methods:It was a cross-sectional study conducted at the department of physiotherapy, University of Lahore Teaching Hospital, Lahore, Pakistan, from July 2020 to August 2020 through non probability-purposive sampling technique. Thirty healthy subjects were tested at thirty-degree lumbar flexion and zero-degree lumbar extension positions. Two raters assessed isometric strength of lumbar flexor and extensor muscles, by a hand-held dynamometer. Strength was measured and recorded by each of the two raters and re-assessed after a week. Correlation and pairwise comparison were done between readings. ICC values were calculated for the assessment of isometric lumbar muscle strength using handheld dynamometer.Results:A total of 30 healthy subjects had participated with mean age of 22.84±1.21 years, height 174.33±6.83 cm, weight 68.58±5.08 kg and BMI 22.52±0.35. Findings showed an excellent intra-rater (ICC 2, k = 0.95 to 0.97) and inter-rater (ICC 2, k = 0.94 to 0.95) reliability.Conclusions:Hand held dynamometer demonstrated an excellent intra- and inter-rater reliability for assessment of isometric lumbar muscles strength of healthy subject at clinical setting as it is simple to use, portable and cost-effective for the precise measurement of lumbar muscles strength.

Epidemiological characteristics of children with coronavirus at a joint commission-accredited hospital in the United Arab Emirates.

Objectives:Our aim was to identify the epidemiological characteristics and transmission patterns of coronavirus (COVID-19) among pediatric patients in the multicultural and multiethnic city of Al Ain in the United Arab Emirates (UAE).Method:A retrospective study was conducted by abstracting data from the electronic medical records of pediatric patients with COVID-19 from two major public hospitals in Al Ain. The data of patients from birth through 16 years of the cases with COVID-19 confirmed by reverse transcription-polymerase chain reaction were obtained. In addition to the epidemiological characteristics, transmission patterns, comorbidities, index cases, travel history, and coinfection with other viruses were analyzed. Cohen's kappa was used to assess interrater reliability and descriptive data. Chi-square test was used to assess significant differences between the variables and was conducted using Statistical Product and Service Solutions software.Results:We identified 298 (150 males, 148 females) laboratory-confirmed cases. The patients’ median age was 7 years. Of these, 50% had parents who also tested positive. Most patients (86.9%) were healthy without any known medical problems. The coinfection rate was ~2%.Conclusion:The epidemiological characteristics of children with COVID-19 in Al Ain are similar to those observed internationally. Children of all ages appeared to be susceptible to COVID-19 and no significant sex or ethnicity differences were detected. Furthermore, this study provides strong evidence of human-to-human transmission.

Prospective Analysis of Fusion Rate with a New Demineralized Bone Matrix

Study background: Autologous Iliac Crest Bone Graft is considered the gold standard graft for use in instrumented spinal fusion procedures. Due to significant morbidity and complication rate, alternate graft materials have been investigated. This study seeks to determine the fusion rate of one such alternative, the K2M VESUVIUSTM Osteobiologic Fibers, a form of demineralized bone matrix. Methods: Prospectively collected CT scans of 27 patients with 29 instrumented levels were taken 1 year post-operatively. These were reviewed blindly by 5 fellowship trained spine surgeons to evaluate for fusion. The fusion mass was graded as no fusion (grade 1), partial unilateral (grade 2), partial bilateral (grade 3), solid unilateral (grade 4), or solid bilateral (grade 5). The fusion mass was then further classified as solid (grade 4-5), probable fusion (grade 3), or non-union (grade 1-2). The Kappa method was used to assess interrater reliability. We calculated the proportion of successful subjects at both cut-off values of either Grade 3 or Grade 4/5. Results: When using a cut-off grade of 3 or higher, 40% of subjects achieved fusion according to 4 of 5 graders and 50% achieved fusion according to 3 of the 5 graders. When using a cut-off grade of 4 or higher, 28% of subjects were found to achieve fusion by at least 4 graders and approximately 50% achieved fusion according to 3 of the graders. When analyzing correlation between the reviewers, 8 of 10 interrater reliabilities demonstrated moderate to good agreement. Correlation by grade showed that only grade 1 had significant agreement amongst the graders with a kappa value of 0.3394. Otherwise, the combined kappa value was 0.1092. Conclusion: Compared to the literature control using iliac crest bone autograft of 70.6% fusion rate, the results with K2M VESUVIUSTM Osteobiologic Fibers DBM were disappointing regardless of the cut off used.

3D quantitative analysis of diffusion-weighted imaging for predicting the malignant potential of intraductal papillary mucinous neoplasms of the pancreas.

PurposeTo investigate the predictors of intraductal papillary mucinous neoplasms of the pancreas (IPMNs) with high-grade dysplasia, using 2-dimensional (2D) analysis and 3-dimensional (3D) volume-of-interest-based apparent diffusion coefficient (ADC) histogram analysis.Material and methodsThe data of 45 patients with histopathologically confirmed IPMNs with high-grade or low-grade dysplasia were retrospectively assessed. The 2D analysis included lesion-to-spinal cord signal intensity ratio (LSR), minimum ADC value (ADCmin), and mean ADC value (ADCmean). The 3D analysis included the overall mean (ADCoverall mean), mean of the bottom 10th percentile (ADCmean0-10), mean of the bottom 10-25th percentile (ADCmean10-25), mean of the bottom 25-50th percentile (ADCmean25-50), skewness (ADCskewness), kurtosis (ADCkurtosis), and entropy (ADCentropy). Diagnostic performance was compared by analysing the area under the receiver operating characteristic curve (AUC). Inter-rater reliability was assessed by blinded evaluation using the intraclass correlation coefficient.ResultsThere were 16 and 29 IPMNs with high- and low-grade dysplasia, respectively. The LSR, ADCoverall mean, ADCmean0-10, ADCmean10-25, ADCmean25-50, and ADCentropy showed significant between-group differences (AUC = 72-93%; p < 0.05). Inter-rater reliability assessment showed almost perfect agreement for LSR and substantial agreement for ADCoverall mean and ADCentropy. Multivariate logistic regression showed that ADCoverall mean and ADCentropy were significant independent predictors of malignancy (p < 0.05), with diagnostic accuracies of 80% and 73%, respectively.ConclusionADCoverall mean and ADCentropy from 3D analysis may assist in predicting IPMNs with high-grade dysplasia.

Enzymatic Debridement for Burn Wound Care: Interrater Reliability and Impact of Experience in Post-intervention Therapy Decision.

Enzymatic debridement (ED) has become a reliable tool for eschar removal. Although ED application is simple, wound bed evaluation and therapy decision post-intervention are prone to subjectivity and failure. Experience in ED might be the key, but this has not been proven yet. The aim of this study was to assess interrater reliability (IR) in post-intervention wound bed evaluation and therapy decision as well as the impact of experience. In addition, the authors introduce video assessment as a valuable tool for post-ED decision-making and education. A video-based survey was conducted among physicians with various experiences in ED. The survey involved multiple-choice and 5-point Likert scale questions about professional status, experience in ED, confidence in post-ED wound bed evaluation, and therapy decision. Subsequently, videos of 15 mixed pattern to full-thickness burns immediately after removal of the enzyme complex were demonstrated. Participants were asked for evaluation of each burn wound, including bleeding pattern and consequent therapy decision. IR ≥ 80% was considered as a consensus. Responses were stratified according to participants’ experience in applying ED (<10, 10–19, 20–49, and ≥50 applications). IR was assessed by chi-square test (raw agreement [RA]; ≥80% was considered as a consensus) and by calculation of Krippendorff’s alpha. In addition, expert consensus for therapy decision was compared with the actual clinical course of each shown patient. Last, participants were asked for their opinion on video as an assessment tool for post-ED wound bed evaluation, decision-making, and training. Thirty-one physicians from 11 burn centers participated in the survey. The overall consensus (raw agreement [RA] ≥ 80%) in post-ED wound bed evaluation and therapy decision was achieved in 20 and 40%, respectively. Krippendorff’s alpha is given by 0.32 (95% confidence interval: 0.15, 0.49) and 0.31 (95% confidence interval: 0.16, 0.47), respectively. Subgroup analysis revealed that physicians with high experience in ED achieved significantly more consensus in post-intervention wound bed evaluation and therapy decision compared with physicians with moderate experience (60 vs 13.3%; P = .02 and 86.7 vs 33.3%; P = .04, respectively). Video analysis was considered a feasible (90.3%) and beneficial (93.5%) tool for post-intervention wound bed evaluation and therapy decision as well as useful for training purposes (100%). Reliability of wound bed evaluation and therapy decision after ED depends on the experience of the rating physician. Video analysis is deemed to be a valuable tool for ED evaluation, decision-making, and user training.

Can the ARAT Be Used to Measure Arm Function in People With Cerebellar Ataxia?

For people with ataxia, there are validated outcome measures to address body function and structure (BFS) impairments and participation; however, no outcome measure exists for upper extremity (UE) activity level in this population. The purpose of this study was to determine whether the action research arm test (ARAT), a measure of UE activity validated for other neurological conditions, might be a useful outcome measure for capturing UE activity limitations in ataxia. A total of 22 participants with ataxia were evaluated to assess construct validity of the ARAT; 19 of the participants were included in the interrater reliability assessment. Participants received a neurologic examination and completed a battery of outcome measures, including the ARAT. ARAT performance was video recorded and scored by 4 additional raters. For construct validity, Spearman rho showed a significant moderate relationship between the ARAT and BSF outcome measures. A small, nonsignificant relationship was noted for the ARAT and the participation measure. For interrater reliability, Spearman rho showed a large, significant relationship among all raters for the ARAT (range = .87-.94). High reliability was demonstrated using the intraclass correlation coefficient ([2,1] = .97). The ARAT is moderately correlated with ataxia BFS outcome measures, but not with participation scores. The ARAT is a measure of UE activity, which is different from BFS and participation outcome measures. The ARAT was identified to have strong interrater reliability among raters with varying amounts of experience administering the ARAT. Thus, for the ataxic population, the ARAT may be useful for assessing UE activity limitations. Ataxia can negatively affect reaching tasks; therefore, it is important to assess UE activity level in people with ataxia. Until this study, no outcome measure had been identified for this purpose. People with ataxia may have difficulty with daily tasks that require reaching. The ARAT is an outcome measure that clinicians can use to assess UE activity limitations to help design a treatment program.

The bumpy road to achieve reliability of clinical profile characteristics in psychosis and related disorders.

ObjectivesProfile characteristics are factors that are relevant for diagnosis, prognosis or treatment. The present study aims to develop a set of clinically relevant profile characteristics. Moreover, our goal is to determine the inter‐rater reliability (IRR) of the selected profile characteristics.MethodsPotential profile characteristics were determined by literature review. Assessment of IRR was done by comparing scores on profile characteristics determined by two researchers. We conducted three subsequent studies: (1) assessment of pre‐training IRR, (2) IRR following implementation of an instruction manual, (3) IRR after optimizing scoring methods. IRR was measured with the Intraclass Correlation Coefficient (ICC).ResultsIRR scores of profile characteristic Illegal activities were high across the three studies (ICC ≥ 0.75). Following training procedures in study 2 and 3, reliability estimates remained low to moderate (ICC < 0.75) for the profile characteristics Support of relatives, Aggression recent and lifetime, substance use and insight recent. IRR scores of the other eight profile characteristics varied from low, moderate to high across studies.ConclusionIRR scores of profile characteristics were highly variable, and mostly inadequate in all three studies. Consequently, further research should focus on specification of severity scores of profile characteristics, optimizing scoring methods and re‐evaluation of IRR.

Reliability and Validity of the Functional Gait Assessment in Incomplete Spinal Cord Injury.

There are limited psychometrically sound measures to assess higher level balance in individuals with incomplete spinal cord injury (iSCI). To evaluate interrater and intrarater reliability and convergent validity of the Functional Gait Assessment (FGA) in individuals with iSCI. Twelve participants (11 male, 1 female) 32 to 73 years old with chronic motor iSCI, American Spinal Injury Association Impairment Scale C (n = 2) or D (n = 10), were included. Participants completed five outcome measures during a single test session including lower extremity motor scores from the International Standards for the Neurological Classification of Spinal Cord Injury, FGA, 10-Meter Walk Test (10MWT), Walking Index for Spinal Cord Injury (WISCI-II), and the Spinal Cord Injury Functional Ambulation Profile (SCI-FAP). Inter- and intrarater reliability for the FGA were excellent. Interrater reliability was excellent with intraclass correlation coefficient (ICC) scores greater than 0.92 (p < .001). Interrater reliability against an expert was also excellent for all raters, with an ICC greater than or equal to 0.92 (p < .01). Intrarater reliability was excellent with an ICC score of greater than 0.91 (p < .002) for all raters. Validity of the FGA with 10MWT was -0.90 (p = .000), FGA with WISCI-II was 0.74 (p = .006), and FGA with SCI-FAP was -0.83 (p = .001). The FGA is a reliable and valid outcome measure to use when assessing gait and balance in individuals with motor iSCI. The FGA provides clinicians with a single tool to utilize across a variety of neurologic diagnoses.