High Intra-rater Agreement Research Articles

Evaluation of exploration time, accuracy, and task difficulty perception in three tactile tests among physiotherapy students

ABSTRACT Objectives Tactile sensitivity is one of the most important skills for developing competence in manual palpation. There is a lack of studies aimed at analyzing the development of tactile sensitivity during different levels of physiotherapy education. The present study aims to compare manual tactile sensitivity in two groups of physiotherapy students. Methods Twenty first-year physiotherapy students (mean age 19.4 yrs ± 0.6) and twenty final-year physiotherapy students (mean age 23.7 yrs ± 3.7) participated in the study. For the tactile sensitivity test, 3 wooden tables were used, in which different geometric structures were engraved. Subjects were instructed to perform a tactile examination of the geometric structures and then reproduce them by drawing on a sheet of paper. The tactile time, drawing time, accuracy, and difficulty of the geometric structures were scored. A two-sample t-test was used for the between-groups comparison if more time in an educational program should result in differences in tactile time, drawing time, accuracy, and difficulty. Linear regression was used to compare the difficulty with the accuracy of all geometric structures. Wilcoxon test was used to test the intra-rater agreement. Results The accuracy of the reproduction of geometric structures 1, 2 and 3 were 77.5%, 27.5% and 45%, for all physiotherapy students respectively. Final-year physiotherapy students spent more time exploring geometric structure 2 (p = 0.014) and geometric structure 3 (p = 0.0018) compared to first-year physiotherapy students. No statistically significant differences were found between groups in drawing time, accuracy, and difficulty of geometric structures. The examiner showed a high intra-rater agreement in the assessment (over 96%). Discussion/conclusion The study showed that the level of education and gained experience during laboratory teaching may be important in improving the palpation skills of physiotherapy students. Tactile sensitivity exercises should be included in physiotherapy education programs.

How reliable is assessment of true vocal cord-arytenoid unit mobility in patients affected by laryngeal cancer? a multi-institutional study on 366 patients from the ARYFIX collaborative group

PurposeIn clinical practice the assessment of the “vocal cord-arytenoid unit” (VCAU) mobility is crucial in the staging, prognosis, and choice of treatment of laryngeal squamous cell carcinoma (LSCC). The aim of the present study was to measure repeatability and reliability of clinical assessment of VCAU mobility and radiologic analysis of posterior laryngeal extension. MethodsIn this multi-institutional retrospective study, patients with LSCC-induced impairment of VCAU mobility who received curative treatment were included; pre-treatment endoscopy and contrast-enhanced imaging were collected and evaluated by raters. According to their evaluations, concordance, number of assigned categories, and inter- and intra-rater agreement were calculated. ResultsTwenty-two otorhinolaryngologists evaluated 366 videolaryngoscopies (total evaluations: 2170) and 6 radiologists evaluated 237 imaging studies (total evaluations: 477). The concordance of clinical rating was excellent in only 22.7% of cases. Overall, inter- and intra-rater agreement was weak. Supraglottic cancers and transoral endoscopy were associated with the lowest inter-observer reliability values. Radiologic inter-rater agreement was low and did not vary with imaging technique. Intra-rater reliability of radiologic evaluation was optimal. ConclusionsThe current methods to assess VCAU mobility and posterior extension of LSCC are flawed by weak inter-observer agreement and reliability. Radiologic evaluation was characterized by very high intra-rater agreement, but weak inter-observer reliability. The relevance of VCAU mobility assessment in laryngeal oncology should be re-weighted. Patients affected by LSCC requiring imaging should be referred to dedicated radiologists with experience in head and neck oncology.

Relation of lumbar intervertebral disc height and severity of disc degeneration based on Pfirrmann scores

Relation of lumbar intervertebral disc height and severity of disc degeneration based on Pfirrmann scores

Three-dimensional reconstruction imaging by C-arm computed tomography accurately visualizes in-stent neointimal formation in patients with stent-assisted coil embolization.

Stent apposition to the vessel wall and in-stent neointimal formation after stent-assisted coil embolization for intracranial aneurysm are important factors associated with postoperative thromboembolic complications. No assessment methods have been established to depict 3-dimensional (3D) all-round in-stent neointimal formation. To demonstrate the superiority of Dyna-3D imaging assessment as a modality for all-round ISNF in comparison with conventional two-dimensional digital subtraction angiography (2D-DSA). Consecutive patients who underwent braided stent-assisted coil embolization for unruptured aneurysm between November 2016 and September 2021 were enrolled. Radiological assessments for stent apposition to the parent vessel after stent deployment and in-stent neointimal formation after 3 months were obtained. Dyna-3D was reconstructed by overlapping a plain image showing stent struts with a rotational DSA image showing the vessel lumen. Reconstructed Dyna-3D images can be rotated to any angle on the screen to evaluate to stent apposition around the vessel and in-stent neointimal formation in 3D, for comparison with 2D-DSA evaluations. Among the 73 patients enrolled, 70 patients (96%) showed complete stent wall apposition on Dyna-3D. Higher intra-rater agreement was confirmed on assessment of in-stent neointimal formation with Dyna-3D (Cohen's κ = 0.811) than with conventional 2D-DSA (Cohen's κ = 0.517). in-stent neointimal formation could not be confirmed on conventional imaging in 9 cases (16%) and on Dyna-3D in 2 cases (3%). The number of in-stent neointimal formations rated as stent wire completely outside the endothelial line was significantly higher with Dyna-3D than with 2D-DSA (p = 0.0001). All-round 3D evaluation by Dyna-3D imaging appears useful for confirming in-stent neointimal formation after braided stent deployment in patients after stent-assisted coil embolization.

Pediatric Rome IV diagnosis agreement is greater than agreement on diagnostic testing

Pediatric Rome IV criteria are used to diagnose childhood functional gastrointestinal disorders (FGIDs). This study of pediatric gastroenterology physicians measured their agreement in (1) Making a pediatric Rome IV FGID diagnosis; and (2) Diagnostic testing for patients with FGIDs. Pediatric gastroenterologists and pediatric gastroenterology fellows at two medical centers completed a survey containing clinical FGID vignettes. For each vignette, raters identified the most likely Rome IV diagnosis(es) and selected which diagnostic test(s) (if any) they typically would obtain. The survey was re-administered within 3months. Inter-rater and intra-rater weighted percent agreement was determined. Linear mixed modeling identified sources of variability in diagnostic testing. Thirty-four raters completed the initial survey of whom thirty-one (91%) completed the repeat survey. Overall inter-rater agreement on Rome IV diagnoses was 68% for initial and repeat surveys whereas intra-rater agreement was 76%. In contrast, overall inter-rater agreement on diagnostic testing was <30% for both initial and repeat surveys and intra-rater agreement was only 57%. Between-physician differences accounted for 43% of the variability in the number of tests selected. Rater identified use of Rome criteria in clinical practice was associated with 1.1 fewer diagnostic tests on average (95% CI 0.2-2.0, p=0.015). Higher intra-rater agreement was noted for diagnostic testing in faculty when compared to fellows (p=0.009). In a multicenter evaluation among pediatric gastroenterology physicians, pediatric Rome IV diagnostic agreement was higher than that reported for previous Rome versions, and higher than agreement on diagnostic testing.

The NASAL score: a novel murine radiologic grading score for sinus disease

The NASAL score: a novel murine radiologic grading score for sinus disease

Intra and inter-rater agreement of inflammatory choroidal neovascular membrane measurements using optical coherence tomography angiography.

Automated measurement algorithm software is not routinely available in optical coherence tomography angiography (OCTA) devices and manual measurement of choroidal neovascular membrane (CNVM) size is necessary. Our aim was to determine intra- and inter-rater agreement of inflammatory CNVM manual measurements obtained with OCTA. OCTA (Triton® Topcon Corporation) images in patients with inflammatory CNVM were imported into ImageJ software v1.50 (NIH image). Two experienced observers performed manual area and perimeter measurements independently, and one of the observers performed the same measurements twice. Agreement was evaluated with intraclass correlation coefficients (ICC) and concordance correlation coefficients (CCC). Bland-Altman plots were plotted to graphically assess concordance. Statistical analysis was performed using STATA v13.0. Sixteen eyes of 16 subjects, with a mean age of 39.0 ± 16.6 years (range 13-71), were included. Mean CNVM area and perimeter was 124.83 ± 117.80 and 4.20 ± 2.00 mm, respectively. Intra-rater ICC for both area and perimeter measured was 0.99 (95% confidence interval (CI) 0.99-0.99). Inter-rater ICC for area and perimeter measured was 0.95 (95%CI 0.87-0.98) and 0.81 (95%CI 0.17-0.94), respectively. Intra-rater CCC for both area and perimeter measured was 0.99 (95%CI 0.99-0.99). Inter-rater CCC for both area and perimeter measured was 0.91 (95%CI 0.81-0.99) and 0.66 (95%CI 0.44-0.88), respectively. Inflammatory CNVM manual measurement showed high intra-rater agreement and moderate inter-rater agreement. Repeatability and reproducibility studies are essential in manual analysis to establish thresholds that can distinguish measurements variation from true clinical change. An automatic algorithm may be helpful to accurately grade lesions and monitor disease activity and response to treatment.

Reliability of and Correlation Between Measurements of Acetabular Morphology.

The authors sought to determine the intra- and interrater reliability of contemporary measures of acetabular morphology among a group of hip surgeons and to determine the correlations between measures of acetabular morphology. On 2 separate occasions, 3 hip surgeons independently performed blinded evaluations of anteroposterior pelvic radiographs of 40 patients. The lateral center-edge angle, medial center-edge angle, acetabular arc, extrusion index, acetabular index, Sharp's angle, posterior wall sign, crossover sign, femoro-epiphyseal acetabular roof index, acetabular wall indexes, and delta angle were assessed. A linear mixed model was used for variance estimation, and kappa and intra-class correlation coefficients were assessed for reliability. Pearson correlation coefficients were calculated for all possible pairs of radiographic measures. The acetabular index had the greatest interrater agreement (0.90; 95% confidence interval [CI], 0.84-0.93), whereas the lateral center-edge angle had the higher intrarater agreement (0.96; 95% CI, 0.92-0.98). The acetabular arc angle had the lowest interrater agreement (0.44; 95% CI, 0.29-0.57), and the Sharp's angle had the lowest intrarater agreement (0.66; 95% CI, 0.41-0.84). The posterior wall sign had an interrater agreement of 0.35 (95% CI, 0.11-0.54) and an intrarater agreement of 0.68 (95% CI, 0.49-0.86). The crossover sign had an interrater agreement of 0.66 (95% CI, 0.43-0.84) and an intrarater agreement of 0.85 (95% CI, 0.52-0.89). The acetabular index, lateral center-edge angle, and extrusion index presented with high coefficients of correlation. In addition, acetabular anteversion correlated with severity of dysplasia. Commonly used parameters such as the acetabular index, lateral center-edge angle, and extrusion index are reliable radiographic parameters to assess acetabular morphology. However, correlation between measures suggests that they may be redundant in quantifying acetabular morphology. [Orthopedics. 2018; 41(5):e629-e635.].

Inter and intra-reader variability in the threshold estimation of auditory brainstem response (ABR) results

Objective: This study aims to examine inter- and intra- reader variability in the ability to estimate hearing threshold through the interpretation of ABR waves of infants. The investigation is based on a large random sample of Australian audiologists.Study design: Sixty-one audiologists were presented with ABR tracings from 15 infants under 12 weeks old. In order to assess inter-rater variability, all cases were presented to all audiologists twice in the same session. Audiologists were asked to nominate a hearing threshold for each of the cases, based purely on their visual judgment of waveforms, and were also asked to answer a brief questionnaire about their experience in ABR testing.Results: Audiologists showed high levels of consistency in reading ABR results. The average inter-reader agreement score was 0.873 and the average intra-reader agreement score was 0.879. Our data show that audiologists’ years of experience in ABR testing significantly correlated with their intra-reader agreement scores (p= 0.004).Conclusion: Audiologists showed a good level of inter- and intra-reader-agreement in ABR interpretation. Years of experience in ABR testing were the main factor that contributed to audiologists’ high intra-rater agreement. Therefore, experience in ABR testing is essential in enhancing the outcome of ABR testing and lowering potential misdiagnosis rates.

Abstract P1-15-12: Development of a photonumeric scale for acute radiation dermatitis in breast cancer patients

Abstract Purpose Scales for rating acute radiation dermatitis (ARD) are inconsistent and have not been validated despite decades of clinical use, making ARD difficult to report reliably. We sought to design a photonumeric scale to consistently describe ARD in breast cancer patients undergoing radiation (RT). Methods Patients undergoing RT for breast cancer were enrolled on a prospective study that included photographs and reporting of physician-rated erythema, hyperpigmentation, and CTCAE toxicity score at baseline and 2, 4, and 6 weeks after initiating RT. Erythema and hyperpigmentation were also quantified using a hand-held colorimetric device. Photographs were taken using a standardized protocol that included 3 views to fully assess the breast/chest wall, axilla, and inframammary fold. 209 photographs from 35 patients with white skin (Fitzpatrick skin types I-IV) and 369 photographs from 50 patients with skin of color (Fitzpatrick skin types V-VI) were clustered according to the apparent severity of ARD. Due to the prevalence of hyperpigmentation that obscured erythema in patients with skin of color, separate images were used to illustrate ARD in this population. Two photonumeric scales (for white skin and skin of color) were developed via an iterative process until group consensus was achieved. Four raters with experience in the evaluation of ARD in breast cancer patients used the photonumeric scale to independently score the entire collection of photographs, sequenced in random order. Intra- and inter-rater agreements were assessed using weighted kappa scores. Results Of the 35 patients with white skin, 20% experienced severe erythema, and 40% experienced dry or moist desquamation. Of the 50 patients with skin of color, 34% experience severe hyperpigmentation, and 48% experienced dry or moist desquamation. Using the photonumeric scales, we observed high intra-rater agreement for independent ratings of erythema or hyperpigmentation (70 to 89% agreement fraction, kappa 0.55 to 0.81) and desquamation (79 to 87% agreement fraction, kappa 0.52 to 0.64). Similarly, we observed moderate to high inter-rater agreement for independent ratings of erythema or hyperpigmentation (61 to 76% agreement fraction, kappa 0.40 to 0.62) and desquamation (69 to 84% agreement fraction, kappa 0.36 to 0.58). Quantitative measurements of erythema in white patients using colorimetry correlated strongly with photonumeric grade (correlation coefficient 0.76, p&amp;lt;0.001), as did physician-rated erythema at the point-of-care (p&amp;lt;0.001). Fitzpatrick score was not significantly associated with maximum photonumeric erythema grade (p = 0.14). Conclusions We report a new photonumeric scale with high intra- and inter-rater reliability for acute radiation dermatitis in breast cancer patients. To our knowledge, this is the first rigorously evaluated scale that is applicable to patients across the spectrum of skin pigmentation, including white skin and skin of color. The photonumeric scale will facilitate consistent reporting of acute radiation dermatitis in research and clinical settings using a simple, standardized instrument. Future work will include prospective real-time clinical validation with multiple raters and correlation with patient-reported outcomes. Funded by a Munn Idea Grant (G011480). Citation Format: Dean Shumway, Eleanor M Walker, Nirav Kapadia, Thy Thy Do, Kent Griffith, Mary Feng, Bonnie DePalma, Reshma Jagsi, Yolanda Helfrich, Erin Gillespie, Alexandria Miller, Adam Liss, Lori J Pierce. Development of a photonumeric scale for acute radiation dermatitis in breast cancer patients [abstract]. In: Proceedings of the Thirty-Seventh Annual CTRC-AACR San Antonio Breast Cancer Symposium: 2014 Dec 9-13; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2015;75(9 Suppl):Abstract nr P1-15-12.

Measurements of bone tunnel size in anterior cruciate ligament reconstruction: 2D versus 3D computed tomography model

BackgroundRevision anterior cruciate ligament (ACL) reconstruction requires a precise evaluation of previous tunnel locations and diameters. Enlargement of the tunnels, despite not usually affecting primary reconstruction outcomes, plays an important role in revision ACL management. Three dimensional (3D) computed tomography (CT) models are reported to be the most accurate method for identifying the tunnel position and possible conflicts with a revision tunnel placement. However, the ability of 3D CT to measure the tunnel size is still not proven. The goal of this study was to evaluate the ability of measuring the size of the bone tunnels in ACL reconstructed knees with 3D CT compared to the traditional two dimensional (2D) CT method.MethodsTwenty-four patients had CT scans performed immediately following ACL reconstruction surgery. Their femoral tunnels size were measured by a standard 2D CT measurement and then compared with three novel 3D CT measuring methods: the best transverse section method, the best fit cylinder method and the wall thickness method. The drill size used during surgery was used as a control measure for the tunnel width. Intra-class correlation coefficients were obtained.ResultsThe intra-class correlation coefficient and respective 95% confidence interval range (ICC [95%CI]) for the three methods compared with the drill sizes were 0.899 [0.811-0.947] for the best transverse section method, 0.745 [0.553-0.862] for the best fit cylinder method, −0.004 [−0.081 to −0.12] for the wall thickness method and 0.922 [0.713-0.97] for the 2D CT method. The mean differences compared to the drill size were 0.02 mm for the best fit transverse section method, 0.01 mm for the best fit cylinder diameter method, 3.34 mm for the wall thickness method and 0.29 mm for the 2D CT method. The intra-rater agreement (ICC [95%CI]) was excellent for the best transverse section method 0.999 [0.998-0.999] and the 2D CT method 0.969 [0.941-0.984].ConclusionsThe 3D best transverse section method presented a high correlation to the drill sizes and high intra-rater agreement, and was the best method for ACL tunnel evaluation in a 3D CT based model.

Reliability of medical record abstraction by non-physicians for orthopedic research

BackgroundMedical record review (MRR) is one of the most commonly used research methods in clinical studies because it provides rich clinical detail. However, because MRR involves subjective interpretation of information found in the medical record, it is critically important to understand the reproducibility of data obtained from MRR. Furthermore, because medical record review is both technically demanding and time intensive, it is important to establish whether trained research staff with no clinical training can abstract medical records reliably.MethodsWe assessed the reliability of abstraction of medical record information in a sample of patients who underwent total knee replacement (TKR) at a referral center. An orthopedic surgeon instructed two research coordinators (RCs) in the abstraction of inpatient medical records and operative notes for patients undergoing primary TKR. The two RCs and the surgeon each independently reviewed 75 patients’ records and one RC reviewed the records twice. Agreement was assessed using the proportion of items on which reviewers agreed and the kappa statistic.ResultsThe kappa for agreement between the surgeon and each RC ranged from 0.59 to 1 for one RC and 0.49 to 1 for the other; the percent agreement ranged from 82% to 100% for one RC and 70% to 100% for the other. The repeated abstractions by the same RC showed high intra-rater agreement, with kappas ranging from 0.66 to 1 and percent agreement ranging from 97% to 100%. Inter-rater agreement between the two RCs was moderate with kappa ranging from 0.49 to 1 and percent agreement ranging from 76% to 100%.ConclusionThe MRR method used in this study showed excellent reliability for abstraction of information that had low technical complexity and moderate to good reliability for information that had greater complexity. Overall, these findings support the use of non-surgeons to abstract surgical data from operative notes.

Reliability of a two-dimensional footprint measurement approach

Reliability of a two-dimensional footprint measurement approach

Intra- and Interrater Agreement with Cumulative Defect Curves

To examine intra- and interrater agreement when analyzing cumulative defect curves. Cumulative defect (Bebié) curves provide a graphical representation of the visual field and allow a subjective classification of diffuse and localized loss. We used 75 Humphrey 30-2 visual field tests, randomly chosen from a database of 782 fields of 113 patients with open-angle glaucoma. Cumulative defect curves were generated and randomly arranged into five sets, with each set containing the 75 curves in a unique sequence. Five raters (two experienced and three inexperienced) rated each set and classified each curve as showing diffuse loss, localized loss, both diffuse and localized loss, or no loss. The intra- and interrater agreement in rating the curves was then analyzed. Intrarater agreement ranged from 73.3-88.0% for perfect rater agreement, with 5 identical ratings. Agreement for experienced versus inexperienced raters gave similar results (means: 84.0% and 77.8%, respectively, for perfect agreement). Interrater agreement for each set, evaluated by the kappa statistic, was substantial for all 5 sets (0.65-0.71). Kappa values for each set were comparable for experienced and inexperienced raters (0.72-0.83 and 0.59-0.69 respectively). Analyzing the nature of visual field loss using the cumulative defect curve is simple to learn and provides high intrarater agreement as well as substantial interrater agreement.