Terms Of Interobserver Agreement Research Articles

Structured reporting in radiologic education - Potential of different PI-RADS versions in prostate MRI controlled by in-bore MR-guided biopsies.

Objectives:To evaluate the efficiency of structured reporting in radiologic education – based on the example of different PI-RADS score versions for multiparametric MRI (mpMRI) of the prostate.Methods:MpMRI of 688 prostate lesions in 180 patients were retrospectively reviewed by an experienced radiologist and by a student using PI-RADS V1 and V2. Data sets were reviewed for changes according to PI-RADS V2.1. The results were correlated with results obtained by MR-guided biopsy. Diagnostic potency was evaluated by ROC analysis. Sensitivity, specificity and correct-graded samples were evaluated for different cutpoints. The agreement between radiologist and student was determined for the aggregation of the PI-RADS score in three categories. The student’s time needed for evaluation was measured.Results:The area under curve of the ROC analysis was 0.782/0.788 (V1/V2) for the student and 0.841/0.833 (V1/V2) for the radiologist. The agreement between student and radiologist showed a Cohen‘s weighted κ coefficient of 0.495 for V1 and 0.518 for V2. Median student’s time needed for score assessment was 4:34 min for PI-RADSv1 and 2:00 min for PI-RADSv2 (p < 0.001). Re-evaluation for V2.1 changed the category in 1.4% of all ratings.Conclusion:The capacity of prostate cancer detection using PI-RADS V1 and V2 is dependent on the reader‘s experience. The results from the two observers indicate that structured reporting using PI-RADS and, controlled by histopathology, can be a valuable and quantifiable tool in students‘ or residents’ education. Herein, V2 was superior to V1 in terms of inter-observer agreement and time efficacy.Advances in knowledge:Structured reporting can be a valuable and quantifiable tool in radiologic education. Structured reporting using PI-RADS can be used by a student with good performance. PI-RADS V2 is superior to V1 in terms of inter-observer agreement and time efficacy.

Grading reliability of the tear film viscosity examination.

To assess the reliability of a tear film (TF) viscosity video grading system. Thirty-four dynamic TF viscosity videos were obtained by a clinically available TF analyzer and objectively sorted according to the movement speed of three arbitrary reflective light particles. A 4-grade system was constructed on a specially designed window for simultaneous comparison with the three standard videos. Two masked graders were invited to grade these videos under a randomized procedure. Observer reliabilities were determined by Spearman's correlation analysis and Bland-Altman agreement analysis. For this four-grade system, the intra-observer correlation was very strong in the two graders (ρ = 0.96 and 0.82; both P < 0.0001). However, the inter-observer correlation showed moderate strength in normal playback speed (ρ = 0.53, P = 0.002 and ρ = 0.52, P = 0.003 for 1st and 2nd gradings, respectively). In slower playback videos, the inter-observer correlation of the two graders was higher (ρ = 0.70 and P < 0.0001) when reduced to 0.8-times playback speed. Moreover, the 0.8-times mode was also significantly better than normal playback mode (P = 0.0204) in terms of inter-observer agreement. The dynamic 4-grade system has an excellent intra-observer reliability and a good inter-observer reliability under 0.8-times playback speed. The grading system established in this study provides a promising solution for rapidly determining the level of TF viscosity.

Chest Computed Tomography (CT) Severity Scales in COVID-19 Disease: A Validation Study.

BackgroundImaging-based quantitative assessment of lung lesions plays a key role in patient triage and therapeutic decision-making processes. The aim of our study was to validate the Total Severity Score (TSS), Chest Computed Tomography Score (CT-S), and Chest CT Severity Score (CT-SS) scales, which were used to assess the extent of lung inflammation in patients with SARS-CoV-2 infection in terms of interobserver agreement and the correlation of scores with patient clinical condition on the day of the study.Material/MethodsA total of 77 chest CT scans collected from 77 consecutive patients hospitalized because of SARS-CoV-2 were included. The scans were assessed independently by 2 radiologists aware of the patients’ positive results of RT-PCR tests. Each chest CT was assessed according to the 3 scales. To assess the interobserver agreement of CT scan assessments, Cohen’s κ and intraclass correlation coefficient (ICC) were calculated.ResultsFor the overall assessment, the κ was 0.944 and the ICC was 0.948 for the TSS; the κ was 0.909 and the ICC was 0.919 for the CT-S; and the κ was 0.888 and the ICC was 0.899 for the CT-SS. The CT-SS (r=0.627 for Radiologist 1 and r=0.653 for Radiologist 2) revealed the strongest positive correlation with the patient clinical condition as expressed using the Modified Early Warning Score.ConclusionsThe interobserver agreement for the 3 evaluated scales was very good. The CT-SS was found to have the strongest positive relationship with the Modified Early Warning Score.

Psychometric Evaluation of the Nursing Outcome Knowledge: Pain Management in People with Chronic Pain.

Pain has a major impact on health and quality of life. Since the level of knowledge of painful conditions can influence how these are addressed and managed, assessing this knowledge in patients becomes crucial. As a result, it is necessary to have culturally adapted and validated instruments that specifically measure patients’ knowledge of chronic pain management. The objective of this study was to carry out the Spanish cultural adaptation and the validation of the outcome Knowledge: Pain Management of the Nursing Outcomes Classification (NOC) in patients with chronic pain, defined as extent of understanding conveyed about causes, symptoms, and treatment of pain. A three-stage study was designed: (1) translation and cultural adaptation through an expert panel, (2) content validation, (3) clinical validation. This study provides nurses with a Spanish version of this scale adapted to their context, as well as a set of structured indicators to measure patients’ knowledge about chronic pain. The results indicated that the culturally adapted Spanish version of the outcome Knowledge: Pain Management had a high level of content validity (CVI = 0.92), with 27 indicators being distributed between two factors. This version has been shown to be reliable in terms of inter-observer agreement (κ = 0.79) and internal consistency (α = 0.95). In conclusion, Knowledge: Pain Management has been shown to be reliable and valid to measure knowledge of chronic pain.

Visual inspection of transvaginal ultrasound videos to characterize uterine peristalsis: an inter-observer agreement study

Contractions in non-pregnant uterine can be assessed by visual inspection of transvaginal ultrasound (TVUS). Many authors have used this method to extract features like contraction frequency and direction. However, visual inspection is a subjective method and the outcome is dependent on the sonographers and video analysts. In this study, we wanted to see which uterine feature is reproducible enough, in terms of inter-observer agreement, to serve as a reliable control for future research. Six observers assessed 80 TVUS videos, and rated video quality, contraction frequency, direction and timing. One observer assessed operating time. A Fleiss' kappa (κ) or an intra-class correlation (ICC) was calculated to determine the inter-observer agreement of all features. The inter-observer agreement in frequency was substantial (ICC = 0.68). Conversely, there was just slight to fair agreement in contraction timing and direction and in video quality: ICC = 0.26, κ = 0.17 and κ = 0.16, respectively. Overall, agreement among technical engineers was better than between medical professionals. The level of agreement was correlated with video quality, phase of the menstrual cycle and individual patient (all χ2 with p < 0.00). The time to analyze one video ranged between 6 and 20min. This study shows that visual inspection of TVUS videos is a fairly reproducible method to assess contraction frequency. However, the operating time is too extensive to implement this method in daily practice. Automated methods could offer a solution for this problem in the future.

International Endocervical Adenocarcinoma Criteria and Classification: Validation and Interobserver Reproducibility.

The current World Health Organization (WHO) classification for endocervical adenocarcinoma (EA) is based on descriptive morphologic characteristics; however, it does not fully reflect our current knowledge of the diverse pathogenesis of cervical glandular neoplasia. A novel classification system, the International Endocervical Adenocarcinoma Criteria and Classification (IECC), which incorporates etiology and biological behavior into the morphologic scheme, has been recently proposed. We aimed to validate the IECC by assessing its interobserver reproducibility in comparison to the WHO system. A cohort of 75 EAs was reviewed independently by 7 gynecologic pathologists and categorized following IECC and WHO criteria based on hematoxylin and eosin material alone and after immunohistochemistry results for p16, PR, p53, Napsin-A, vimentin, CDX2, and GATA3 were provided. Human papillomavirus (HPV) in situ hybridization and polymerase chain reaction results were compared with consensus diagnoses. IECC was superior to WHO in terms of interobserver agreement with κ=0.46 versus 0.3, respectively, on hematoxylin and eosin review and κ=0.51 versus 0.33, respectively, with immunohistochemistry. Under the IECC, 73 (97%) of EAs had majority agreement (≥4 reviewers in agreement) whereas 42 (56%) had perfect agreement (7/7 reviewers in agreement). Conversely, WHO showed majority agreement in 56 (75%) and perfect agreement in only 7 (10%) EAs. Reproducibility was poor in HPV-related WHO types (usual κ=0.36, mucinous not otherwise specified κ=0.13, intestinal κ=0.31, villoglandular κ=0.21) and good in major HPV-unrelated categories (gastric type κ=0.63, clear cell κ=0.81, mesonephric κ=0.5). Classification as per the IECC had excellent correlation with HPV status (by RNA in situ hybridization or polymerase chain reaction). We have shown that the IECC has superior interobserver agreement compared with the WHO classification system, and that distinction between HPV-related and HPV-unrelated EA can be made with good reproducibility and excellent prediction of HPV status. WHO morphologic variants of HPV-related EA are poorly reproducible. Conversely, agreement is high among important high-risk HPV-unrelated subtypes. Thus, our results further support replacing the current WHO classification with the IECC.

Added value of prone CT in the assessment of honeycombing and classification of usual interstitial pneumonia pattern

ObjectiveTo retrospectively investigate whether prone CT improves identification of honeycombing and classification of UIP patterns in terms of interobserver agreement and accuracy using pathological results as a reference standard. Materials and methodsInstitutional review board approval with waiver of patients’ informed consent requirement was obtained. HRCTs of 86 patients with pathologically proven UIP, NSIP and chronic HP between January 2011 and April 2015 were evaluated by 8 observers. Observers were asked to review supine only set and supine and prone combined set and determine the presence of honeycombing and UIP classification (UIP, possible UIP, inconsistent with UIP). The diagnosis was regarded as correct when UIP pattern on CT corresponded to pathological UIP. ResultsInterobserver agreement of honeycombing identification among radiologists was only fair on the supine and combined set (weighted κ=0.31 and 0.34). Additional review of prone images demonstrated a significant improvement in interobserver agreement (weighted κ) of UIP classification from 0.25 to 0.33. Prone CT conferred a significant improvement in interobserver agreement of UIP classification for trainee radiologists (from 0.10 to 0.34) while no improvement was found for board-certified radiologists (from 0.35 to 0.31). There were no significant differences in the accuracy of UIP pattern with reference to pathological results between the supine and combined set (78.8% (145/184) and 81.3% (179/220), P=0.612). ConclusionAdditional review of prone CT can improve overall interobserver agreement of UIP classification among radiologists with variable experiences, particularly for less experienced radiologists, while no improvement was found in honeycombing identification.

Interobserver agreement of contrast-enhanced harmonic endoscopic ultrasonography in the evaluation of solid pancreatic lesions.

Background and study aims: Previous reports assessing the reproducibility of contrast-enhanced harmonic endoscopic ultrasonography (CH-EUS) in the evaluation of solid pancreatic lesions (SPLs) involved mainly experienced endosonographers. We aimed to assess the interobserver agreement (IOA) of CH-EUS in the evaluation of SPLs by endoscopists with different levels of experience in EUS and CH-EUS. Participants and methods: A cross-sectional observational multicenter study was designed and included 11 endoscopists who were divided into four groups according to their experience in EUS and CH-EUS: group A (long experience in EUS and CH-EUS); group B (short experience in EUS and CH-EUS); group C (long experience in EUS and no experience in CH-EUS); and group D (no experience in EUS or CH-EUS). The observers independently classified the patterns of 60 CH-EUS video sequences of 60 SPLs after a 20-minute training session. For each group, we calculated the IOA (kappa statistic, κ) of CH-EUS and the accuracy of CH-EUS for the diagnosis of pancreatic adenocarcinoma by comparing the pattern of CH-EUS indicative of pancreatic adenocarcinoma (hypo-enhanced contrast pattern) with the final diagnosis. Results: The overall IOA for CH-EUS was fair (κ = 0.32; 95 %CI 0.22 – 0.41). Group A (κ = 0.63; 95 %CI 0.45 – 0.85) had the highest IOA, followed by group C (κ = 0.54; 95 %CI 0.39 – 0.71), group B (κ = 0.38; 95 %CI 0.22 – 0.55), and group D (κ = 0.21; 95 %CI 0.07 – 0.36). The IOA of groups A and C was significantly higher than that of group D. No significant difference was seen between groups A, B, and C or between groups B and D in terms of IOA. Group A (area under the curve under summary receiver operating characteristic [AUROC] = 0.67; 95 %CI 0.58 – 0.75) had the highest accuracy for the diagnosis of pancreatic adenocarcinoma, followed by group C (AUROC = 0.58; 95 %CI 0.50 – 0.65), group B (AUROC = 0.55; 95 %CI 0.48 – 0.63), and group D (AUROC = 0.51; 95 %CI 0.43 – 0.58). The diagnostic accuracy of group A was not significantly different from that of group C, but it was significantly higher than that of groups B and D. No significant difference was seen between groups B, C, and D in terms of diagnostic accuracy. Conclusions: CH-EUS is reproducible in the evaluation of SPLs, even between endoscopists with no or limited experience in EUS and/or CH-EUS. Long experience in EUS is a major contributor to the IOA and diagnostic accuracy of CH-EUS.

The Pauwels classification for intracapsular hip fractures: Is it reliable?

The Pauwels classification for the femoral neck fracture is still broadly used in literature and clinical practise. However, this classification has never been tested for its reliability in terms of inter-observer agreement. We assessed whether or not it is reliable to use the Pauwels classification in pre-operative planning. Ten observers classified 100 intra-capsular femur fractures. The inter-observer agreement was calculated using the multi-rater Fleiss’ kappa. The Pauwels classification showed an inter-observer agreement of κ0.31 (0.01). Classification of intra-capsular hip fractures according to the Pauwels classification using the Pauwels angle is unreliable and its use should be avoided.

Radiographic signs of scaphoid union after bone grafting: The analysis of inter-observer agreement and intra-observer reproducibility

The diagnosis of radiological union of scaphoid bone after bone grafting requires clear evidence of bony trabeculae traversing the graft from the proximal to the distal pole on at least two of four standard scaphoid views. This sign is the only objective assessments of union. Radiographs of the scaphoid taken 18 weeks after operation, however, can be difficult to interpret. This fact led us to question whether radiographs of scaphoid at 18 weeks provide reliable and objective indication of union. Our study was, therefore, designed to determine the reliability of the radiographic diagnosis of scaphoid union after bone grafting by testing the degree of inter-observer agreement and reproducibility. Out of 30 sets of the scaphoid bone radiographs after bone grafting taken 18 weeks after operation, 15 of good quality were selected. Each set included four views: postero-anterior, lateral, semi- pronated and semi-supinated. Seven observers were tested: three orthopedic consultants, three residents and one consultant in radiology. Each was presented with 15 sets of radiographs designated from 1 to 15 and each was asked to answer the question: "Are there trabeculae crossing the fracture site?" Possible answers were 'yes' or 'no'. Eight weeks later, the same 15 sets of radiographs were marked in alphabetic order from A to K and presented to the same seven observers. Data was then analyzed and expressed in terms of interobserver agreement in pairs and intra-observer reproducibility. Calculation was done by kappa statistics so that the degree of disagreement was taken into account and allowance was made for chance agreement. Kappa values can vary from -1.0 (complete disagreement) through 1 (chance agreement) to +1 (complete agreement). For all 15 sets of radiographs, the degree of agreement between each pair of observers was illustrated in Table 2. It demonstrated the level of agreement between each pairs of seven observers. The overall median kappa coefficient for inter-observer agreement was 0,46. This value corresponded to "moderate" strength of agreement. Median kappa coefficient for consultant was 0.62, but median kappa coefficient for residents was 0.43. The degree of agreement between the opinions of each observer at his first and at his subsequent reviews of the same set of radiograph after 8 weeks was presented in Table 3. The overall median Kappa coefficient for the intraobserver agreement for all seven observers was 0.54. The ability of seven observers to reproduce their own opinion regarding trabeculae crossing the fracture line was poor to use it for assessment of bone union. Our conclusion is that radiographs taken 18 weeks after scaphoid grafting cannot be reliable and reproducible for assessment of bone union.

Developing a General Outcome Measure of Growth in Movement for Infants and Toddlers

Proficiency in movement, an important outcome in early childhood, is necessary for physical, cognitive, and social—emotional development. The development of an experimental measure for assessing growth in movement in children ages birth to 3 years is described. Based on general outcome measurement (GOM) procedures (e.g., Deno, 1997), the measure was intended for the identification of children having difficulty acquiring movement skills and the evaluation of the effectiveness of interventions for these children. Results from its use with a sample of 29 infants and toddlers demonstrated the feasibility of the measure. The 6-minute GOM was found to be reliable in terms of interobserver agreement and odd—even and alternate forms indices. Adequate criterion validity was demonstrated through use of a standardized measure of motor abilities and caregiver ratings of children's movement skills. The GOM was sensitive to changes in key skill elements and growth in total movement rate with age. This Movement GOM holds great promise as a quick, inexpensive, and readily interpretable means of monitoring growth in movement for children from birth to 3 years.

The Critical Nursing Situation Index for safety assessment in intensive care.

The assessment of critical nursing situations can be a valuable tool in the detection of weak elements in the safety of patients and the quality of care in the ICU. A critical nursing situation can be defined as any observable situation, which deviates from good clinical practice and which may potentially lead to an adverse event. The aim of our study was to establish the feasibility, reliability and validity of the Critical Nursing Situation Index (CNSI) as a tool for assessing the safety and the quality of nursing in the ICU. We described the deviations from standards and protocols in daily ICU nursing care, selected those with an implicit, clear risk for the patients and translated them into explicitly observable items. If an item was applicable during observation of the ICU practice, a critical nursing situation could be recorded as either true or false. The reliability of the CNSI was defined in terms of inter-observer agreement. The validity was assessed by exploring the relationship between the nursing time available (more or less than 30 min per patient per hour) and the incidence of critical nursing situations. The study was performed in the ICU of a teaching hospital (30 IC beds) in which all disciplines, including cardiothoracic surgery and neurosurgery, were represented. The CNSI was randomly applied to 83 ICU patients over a period of 3 months (200 times). The reliability of the index was substantial (Kappa values in the range > or =0.70 to > 0.80). In terms of validity, less nursing time resulted in more critical situations (pooled relative risk (RR) 1.36; 95% confidence limits 1.11/1.67). The CNSI is simple to use and has encouraging metric properties, whereas the assessments are closely related to direct patient care. Moreover, the CNSI provides a tool for safety assessment by monitoring potentially dangerous situations that are generally regarded as needing to be avoided.

Quantification of pyuria by two methods correlation and interobserver agreement.

The aim of this study was to correlate the results of two methods of quantifying pyuria and to compare the diagnostic reliability of the two methods in terms of interobserver agreement. Three independent observers counted leucocytes in 56 urine samples by using high-power-field (HPF) microscopy on centrifuged urine sediment and by using a Bürker haemocytometer on uncentrifuged urine. The commonly accepted diagnostic level of ten leucocytes/mm3 in the haemocytometer was found to be equivalent to two leucocytes/HPF. This agrees with other reports but contrasts with the usual notion of a diagnostic level of five to ten leucocytes/HPF, and implies that urinary tract infection in the presence of symptoms should not be ruled out when less than five leucocytes/HPF are found. Kappa statistics were used to evaluate the interobserver agreement. The observers agreed on the diagnosis in 96% (K = 0.92) of the urines when the haemocytometer was used, and in 93-96% (K = 0.81-K = 0.91) when leucocytes were counted per HPF. The differences in Kappa values were not statistically significant. Our material indicates that the method of quantifying pyuria in centrifuged urine sediment by HPF microscopy may be used as the routine method in general practice.

Item reliability for the Griffiths scales of mental development.

As a preliminary to the revision of the Griffiths scales, the 498 items constituting the original test were studied in terms of inter-observer agreement over the scoring, using videotapes of tests, given to children aged from 2 months to 7 years, and large scoring panels. Fifty-seven items failed to be administered or scored adequately often. One hundred and twelve were found to be unreliable on one in three administrations and only 23 were unreliable on two of three administrations. Among items scored sufficiently often, 88% of all administrations were reliable. Subscale differences were not as expected since the locomotor, personal--social and hearing and speech scales fared no worse than hand--eye coordination. Inter-observer agreement varied with the age of the children. Subscale weaknesses were examined in terms of the age ranges most concerned.

The reliability of self-recorded behaviors

The purpose of this paper is to make explicit some of the problems involved in assessing the reliability of self-recorded measures of behavior. An examination of investigations in the area of self-regulation of behavior revealed instances in which there either had been no assessment of the reliability of the self-recordings or else such procedures were not made explicit. In the absence of reliability estimates it is difficult to evaluate the reported successes of behavior-therapy techniques that make use of self-control procedures. At the very least, reliability estimates in terms of interobserver agreement should be reported. The current procedures for recording self-behavior and their relationship to the problems of reliability assessment are discussed.