Interobserver Reliability Study Research Articles

Development of a mobile application to represent food intake in inpatients: dietary data systematization

BackgroundNutritional risk situations related to decreased food intake in the hospital environment hinder nutritional care and increase malnutrition in hospitalized patients and are often associated with increased morbidity and mortality. The objective of this study is to develop and test the reliability and data similarity of a mobile application as a virtual instrument to assess the acceptability and quality of hospital diets for inpatients.MethodsThis intra- and interobserver development and reliability study investigated an in-hospital food intake monitoring application based on a validated instrument for patients with infectious diseases who were treated at the Evandro Chagas National Institute of Infectious Diseases (INI/FIOCRUZ). The instrument was sequentially administered to patients 48 h after admission to INI hospital units using the printed instrument (paper) and the digital application (ARIETI) simultaneously. The tested reliability factor was the consistency of the method in the digital platform, checking whether the application provided equivalent data to the paper instrument, and finally, a statistical analysis plan was performed in the R platform version 4.2.0. This project was authorized by the FIOCRUZ/INI Research Ethics Committee.ResultsThe ARIETI was developed and tested for reliability in 70 participants, showing a similar ability to calculate caloric intake in Kcal, protein intake (g), the proportion of caloric intake and protein intake relative to the prescribed goal. These instrument comparison analyses showed statistical significance (p < 0.001). The application was superior to the paper-based instrument, accelerating the time to perform the nutritional risk diagnosis based on food intake by approximately 250 s (average time).ConclusionsThe ARIETI application has demonstrated equivalent reliability compared to the original instrument. Moreover, it optimized the time between the diagnosis of nutritional risk related to dietary intake and the nutritionist’s decision making, showing an improved ability to maintain information quality compared to the paper-based instrument.

Reproducibility of the Mayo and Schatzker classification systems in proximal ulna fractures

IntroductionA fracture classification system should provide a reliable and reproducible means of communication between different parties. It should be logical and understandable, with few categories to memorize. The aim of this study was to determine the intra- and interobserver reliability of the Schatzker and Mayo classification systems for the assessment of proximal ulna fractures. Materials and methodsIntra- and interobserver reliability studies were conducted on 39 X-rays of injured elbows drawn randomly from 74 cases previously used in a series on predictors of ulnohumeral osteoarthritis in proximal ulna fractures. Ten observers independently reviewed these X-rays on 2 separate occasions 3 months apart. The fracture type was assessed according to the Schatzker and Mayo classification systems during each reading session. Cohen's and Fleiss’ kappa were used to measure the intra- and interobserver reliability. ResultsThe Schatzker classification had a fair interobserver reliability for the first (Schatzker R1, Fleiss’ κ: 0.394) and second (Schatzker R2, Fleiss’ κ: 0.351) readings. The mean intraobserver reliability value between the 10 reviewers for the Schatzker classification was rated as substantial (0.61). The Mayo classification had a fair interobserver reliability for the first (Mayo R1, Fleiss’ κ: 0.278) and second (Mayo R2, Fleiss’ κ: 0.292) readings. The mean intraobserver reliability value between the 10 reviewers for the Mayo classification was rated as fair (0.52). DiscussionThe classification systems for proximal ulna fractures showed poor reproducibility between the different observers since they had low interobserver agreement values. Nevertheless, their use remained reliable since the measured intraobserver agreement value was deemed substantial. Level of evidenceIV; retrospective.

Kaleidos-based binocular vision analyzer for the estimation of refractive errors and horizontal ocular deviation in children

ABSTRACT Clinical relevance Knowing the agreement and interobserver variability of the 2WIN-S photorefractometer with cycloplegic retinoscopy and prism cover test is important for judging the clinical usefulness of the device. Background This study aims to determine an agreement and interobserver reliability between 2WIN-S photorefractometer and cycloplegic retinoscopy for estimating refractive errors and between 2WIN-S and prism cover test for measuring horizontal ocular deviation in children. Methods Two groups of 129 and 64 participants (range, 5 to 16 years) were recruited for the agreement and interobserver reliability studies, respectively. Non-cycloplegic 2WIN-S refraction was compared with cycloplegic retinoscopy for low and moderate myopia and hyperopia. Similarly, corneal reflex-function (CR-function) of 2WIN-S was compared with prism cover test for horizontal ocular deviation. Bland-Altman plots were used to depict agreement, and the intraclass correlation coefficient (ICC) was used for determining interobserver reliability of repeated measures. Results The mean differences in spherical equivalent, J0 (Cartesian astigmatism) and J45 (oblique Jackson cross-cylinder) between cycloplegic retinoscopy and 2WIN-S were 1.10 (95% CI = 0.97 to 1.21, p < 0.001), −0.06 (95% CI = −0.09 to −0.03, p < 0.001) and 0.009 (95% CI = −0.01 to 0.03, p = 0.35), respectively. The overall median difference in the horizontal ocular deviation between prism cover test and 2WIN-S was −4.5 (Interquartile range = −12.0 to 4.0). For constant strabismus, this difference was insignificant (median= −2.0, interquartile range = −11.0 to 6.7, p = 0.19) between prism cover test and 2WIN-S. The ICC was high for the spherical equivalent (0.997; p < 0.001), J0 (0.932; p < 0.001) and J45 (0.901; p < 0.001), whereas it was low-to-moderate for horizontal ocular deviation (0.50; p < 0.001). Conclusion 2WIN-S refraction was in agreement with cycloplegic retinoscopy for low and moderate myopia and hyperopia, and requires an adjustment for the consistent differences between cycloplegic retinoscopy and 2WIN-S refraction. The measurement of constant strabismus with 2WIN-S is comparable to that of prism cover test.

Ultrasound Strain Elastography Reliability in the Assessment of the Plantar Fascia and Its Relationship with the Plantar Thickness in Healthy Adults: An Intra and Interobserver Reliability Study in Novice Evaluators.

This study was aimed at verifying both the intraobserver and interobserver reliability of measuring plantar fascia stiffness for a given image in healthy active adults. This study is reported following the Guidelines of Reporting Reliability and Agreement Studies. A total of 20 plantar fascia from healthy volunteers were analyzed. The thickness of the plantar fascia was measured vertically from the anterior edge of the inferior calcaneal border to the inferior border of the plantar fascia and the ultrasound elastography measurement was taken at the calcaneal insertion of the plantar fascia with the region of interest one centimeter away from the insertion. The ultrasound strain elastography measurements: the right intraobserver 1 showed an ICC value of 0.9 and the left intraobserver 1 showed an ICC value of 0.78, while the right intraobserver 2 showed an ICC value of 0.91 and the left intraobserver 2 showed an ICC value of 0.83. Interobserver measurements showed excellent reliability with a right ICC value of 0.8 and a left ICC value of 0.9 for the plantar fascia thickness measurements. The results of this study showed a strong correlation between left and right plantar fascia thickness. The intraobserver reliability was excellent for both plantar fascia ultrasound strain elastography and thickness measurements, with interobserver measurements showing excellent reliability.

Diagnostic Value of Diffusion-Weighted Imaging with Background Body Signal Suppression (DWIBS) for the Pre-Therapeutic Loco-Regional Staging of Cervical Cancer: A Feasibility and Interobserver Reliability Study.

(1) Background: cervical cancer is one of the leading causes of cancer-related deaths and the fourth most common cancer among women worldwide. Magnetic resonance imaging (MRI) is the modality of choice for loco-regional staging of cervical cancer in the primary diagnostic workup beginning with at least stage IB. (2) Methods: we retrospectively analyzed 16 patients with histopathological proven cervical cancer (FIGO IB1−IVA) for the diagnostic accuracy of standard MRI and standard MRI with diffusion-weighted imaging with background body signal suppression (DWIBS) for the correct pre-therapeutic assessment of the definite FIGO category. (3) Results: In 7 out of 32 readings (22%), DWIBS improved diagnostic accuracy. With DWIBS, four (13%) additional readings were assigned the correct major (I−IV) FIGO stages pre-therapeutically. Interobserver reliability of DWIBS was weakest for parametrial infiltration (k = 0.43; CI-95% 0.00−1.00) and perfect for tumor size <2 cm, infiltration of the vaginal lower third, infiltration of adjacent organs and loco-regional nodal metastases (k = 1.000; CI-95% 1.00−1.00). (4) Conclusions: the pre-therapeutic staging of cervical cancer has a high diagnostic accuracy and interobserver reliability when using standard MRI but can be further optimized with the addition of DWIBS sequences when reporting is performed by an experienced radiologist.

Sagittal Plane Assessment in Deformity Correction Planning: The Sagittal Joint Line Angle.

Evaluate the validity of a recent approach to calculate the knee flexion or extension contracture contributing to the overall sagittal deformity using the sagittal mechanical axis angle (SMAA) for the overall alignment assessment and sagittal joint line angle (SJLA) for soft tissue contribution. The methods of evaluating these angles and their clinical applications are discussed. In total, 107 normal limbs met the criteria and were divided into two groups: skeletally mature and immature. Sagittal alignment was evaluated using the Bone Ninja iPad application, and the posterior distal femoral angle (PDFA), posterior proximal tibial angle (PPTA), SMAA and SJLA were recorded. In skeletally immature patients, mean SJLA was 13.46° [standard deviation (SD), 4.55°], and in mature patients, it was 16.91° (SD, 2.948°). The PDFA and PPTA were consistent with previously published measurements. The SJLA method is a practical way to quantify the soft tissue contribution and degree of contracture. It can also be used for monitoring deterioration or improvement of knee range of motion during lengthening or physical therapy. All patients in this study presented to our clinic with symptoms on the contralateral side. This, in addition to the retrospective nature, was a limitation in our study.We recommend a validity study to compare our SJLA method to the classic anterior cortical line angle (ACL) method in addition to an inter-observer and intra-observer reliability study for the SJLA. We also recommend a study on completely normal asymptomatic subjects to better standardise the angle measurements in skeletally immature patients at different ages. Abalkhail TB, McClure PK. Sagittal Plane Assessment in Deformity Correction Planning: The Sagittal Joint Line Angle. Strategies Trauma Limb Reconstr 2022;17(3):159-164.

Additional Diffusion-Weighted Imaging with Background Body Signal Suppression (DWIBS) Improves Pre-Therapeutical Detection of Early-Stage (pT1a) Glottic Cancer: A Feasibility and Interobserver Reliability Study.

(1) Background: Early-stage glottic cancer is easily missed on magnetic resonance imaging (MRI). Diffusion-weighted imaging (DWI) may improve diagnostic accuracy. Therefore, our aim was to assess the value of adding diffusion-weighted imaging with background body signal suppression (DWIBS) to pre-therapeutic MRI staging. (2) Methods: Two radiologists with 8 and 13 years of experience, blinded to each other's findings, initially interpreted only standard MRI, later DWIBS alone, and afterward, standard MRI + DWIBS in 41 patients with histopathologically proven pT1a laryngeal cancer of the glottis. (3) Results: Detectability rates with standard MRI, DWIBS only, and standard MRI + DWIBS were 68-71%, 63-66%, and 73-76%, respectively. Moreover, interobserver reliability was calculated as good (κ = 0.712), very good (κ = 0.84), and good (κ = 0.69) for standard MRI, DWIBS only, and standard MRI + DWIBS, respectively. (4) Conclusions: Standard MRI, DWIBS alone, and standard MRI + DWIBS showed an encouraging detection rate, as well as distinct interobserver reliability in the diagnosis of early-stage laryngeal cancer when compared to the definitive histopathologic report.

Usefulness of digital tomosynthesis in diagnosing cervical ossification of the posterior longitudinal ligament: a comparative study with other imaging modalities.

The diagnosis and classification of ossification of the posterior longitudinal ligament (OPLL) can be difficult with radiography alone; therefore, computed tomography (CT) is also usually performed. There are many reports on the usefulness of digital tomosynthesis (DTS) for image analysis in orthopedics. This study aimed to compare the accuracy of DTS with radiography and CT for the diagnosis and classification of cervical OPLL (C-OPLL). We included 31 patients with OPLL and 30 with cervical spondylotic myelopathy. The patients' cervical spine radiography, DTS, and CT images were each evaluated twice by three specialists and three residents. In the intra-observer reliability study, there was one observer with a fair level of kappa values for radiography and DTS among three residents. The kappa values for CT were the best for all observers. In the inter-observer reliability study, the interclass correlation coefficient (ICC) values were high for both diagnosis and classification by specialists at the almost perfect level for all three imaging modalities. On the other hand, the ICC values for both diagnosis and classification for radiography by the residents were lower than those for DTS and CT. This study revealed that DTS may be an alternative to CT for the diagnosis and classification of C-OPLL by specialists. Caution should be exercised in diagnosing and classifying C-OPLL using radiography and DTS by residents, and the use of CT is recommended.

Is the Diagnosis of 5th Metatarsal Fracture Type Consistent? An Inter-Observer Reliability Study

Category: Midfoot/Forefoot; Trauma Introduction/Purpose: Treatment pathways of 5th metatarsal fractures are commonly directed based on fracture classification, with Jones types for example, requiring closer observation and possibly more aggressive management. Our primary objective was to investigate the reliability of assessment of subtypes of 5th metatarsal fractures by different observers. Methods: Patients were identified from our prospectively collected database. We included all patient referred to our virtual fracture clinic with a suspected or confirmed 5th metatarsal fracture. Plain AP radiographs were reviewed by two observers, who were initially trained on the 5th metatarsal classification identification. Zones were defined as Zone 1.1, 1.2, 1.3, 2, 3, diaphyseal shaft (DS), distal metaphysis (DM) and head. Patients were excluded from analysis if neither observer could identify a fracture or if one or more observers could not access the images. An inter-observer reliability analysis using Cohen's Kappa coefficient was carried out, and degree of observer agreement described using Landis &amp; Koch's description. All data was analysed using IBM SPSS v.27. Results: 1360 patients who met the inclusion criteria were identified. The two observers had moderate agreement when identifying fractures in all zones, apart from zone 1.2 and DM fractures, which only achieved 'fair' agreement (K=.308 and .381 respectively). Zones 1.1 (K=.538), 1.3 (K=.511), 2 (K=.515), 3 (K=.57), DS (K=.545) and Head (K=.558) all tended towards but did not achieve substantial agreement. Slight agreement with the next proximal and distal adjacent zones were found in Zones 1.1, 1.2 and 1.3. Slight agreement with the adjacent distal zone was found in Zone 2 and DM fractures. Zone 3 has slight agreement with Zone 2 proximally and there is an apparent difficulty with distal DS fractures resulting in a lot of crossover with DM, achieving a fair level of agreement (DS 312 vs 196; DM 120 vs 237; K=.398, p&lt;.001). Conclusion: Reliability of sub-categorising 5th metatarsal fractures using standardised instructions conveys moderate agreement in most cases. If the region of the fracture is going to be used in an algorithm to guide a management plan and clinical follow up during a virtual clinic review, defining fractures of zones 1-3 needs careful consideration.

Inter- and intra-observer reliability of BITSS scale in the diagnosis and early follow-up of infectious diarrhea in children under 3 years of age

The BITSS scale (Brussels Infant and Toddler Stool Scale) is an instrument used in the diagnosis and follow-up of infants and young children with infectious diarrhea. It consists of 7 pictures of diapers with different types of stools. To determine intra and interobserver reliability of the BITSS scale in infants and children using diapers. Reliability study. Three pediatri cians with experience in treating childhood infectious diarrhea were selected and, after training in applying the BITSS scale, they evaluated 70 pictures of stools obtained from pediatric patients, repre senting the 7 different types of stools categorized by the scale (10 images for each item). Two pediatri cians participated in the interobserver reliability study and performed a blind evaluation of the same pictures, assigning a value to each one, and the third pediatrician participated in the intra-observer analysis, evaluating 70 images on two different occasions, 15 days apart. Interobserver agreement and kappa statistics were determined for the scale and each scale item. 280 measurements were made, 140 for the inter-observer reliability study and 140 for the intra-observer one. The degree of inter-observer agreement for the scale was 85.7%; with a kappa of 0.817 (95% CI 0.72; 0.92), and standard error of 0.05; and the intra-observer kappa for the scale was 0.883 (95% CI 0.80; 0.96), with a standard error of 0.04. The BITSS scale has good intra- and interobserver reliability.

P.146 Non-contrast CT markers of intracerebral hemorrhage expansion: a predictive accuracy and reliability study

Background: We evaluated (1) the predictive accuracy and (2) multi-observer reliability of non-contrast CT markers of hematoma expansion (HE). Methods: In 124 patients with spontaneous intracerebral hemorrhage, two investigators documented the presence of six density (Barras density, hypodensity, black hole, swirl, blend, fluid level) and three shape (Barras shape, island, satellite) expansion markers, with discrepancies resolved by a third rater. We defined HE as any one of (1) &gt;6 mL absolute or &gt;33% relative growth of the intraparenchymal hematoma or (2) an absolute growth of &gt;1 mL or new development of intraventricular hematoma. A subsample of 60 patients was used for the inter-observer reliability study in 13 raters. Seven raters participated in the intra-rater study. Results: The sensitivity of markers for HE varied between 4% (fluid level) and 78% (satellite), while specificity ranged from 37% (swirl) to 97% (black hole). Almost perfect inter-rater agreement was observed for the swirl (0.89) and fluid level (0.83) markers, while hypodensity (0.65) showed substantial agreement. Only the blend and fluid level markers achieved substantial intra-rater agreement (&gt; 0.6) in all raters. Conclusions: Non-contrast CT markers of HE showed lower reliability and predictive accuracy than previously reported. Future studies should address means to improve NCCT-based HE prediction.

Comparing deep learning-based automatic segmentation of breast masses to expert interobserver variability in ultrasound imaging

Deep learning is a powerful tool that became practical in 2008, harnessing the power of Graphic Processing Unites, and has developed rapidly in image, video, and natural language processing. There are ongoing developments in the application of deep learning to medical data for a variety of tasks across multiple imaging modalities. The reliability and repeatability of deep learning techniques are of utmost importance if deep learning can be considered a tool for assisting experts, including physicians, radiologists, and sonographers. Owing to the high costs of labeling data, deep learning models are often evaluated against one expert, and it is unknown if any errors fall within a clinically acceptable range. Ultrasound is a commonly used imaging modality for breast cancer screening processes and for visually estimating risk using the Breast Imaging Reporting and Data System score. This process is highly dependent on the skills and experience of the sonographers and radiologists, thereby leading to interobserver variability and interpretation. For these reasons, we propose an interobserver reliability study comparing the performance of a current top-performing deep learning segmentation model against three experts who manually segmented suspicious breast lesions in clinical ultrasound (US) images. We pretrained the model using a US thyroid segmentation dataset with 455 patients and 50,993 images, and trained the model using a US breast segmentation dataset with 733 patients and 29,884 images. We found a mean Fleiss kappa value of 0.78 for the performance of three experts in breast mass segmentation compared to a mean Fleiss kappa value of 0.79 for the performance of experts and the optimized deep learning model.

Reliability study of radiology interpretation in osteosarcoma

Introduction: Radiography interpretation of osteosarcoma is almost similar with the benign and other malignant bone tumors. The radiologists’ ability to accurately diagnose osteosarcoma will determine the therapy and diagnostic consistency. Currently, there is lack of evidence on radiography interpretation reliability of osteosarcoma performed by radiologists. This study aimed to evaluate the interobserver and intraobserver radiologists’ compatibility to confirm the diagnosis of osteosarcoma and explore the interpretation variability of radiological description.Methods: This was an interobserver and intraobserver reliability study for osteosarcoma radiograph interpretation. The osteosarcoma radiographs, which have been histopathologically diagnosed, were evaluated retrospectively by two observers and twice by one observer in a different timeline. The components of osteosarcoma identified through the radiographs are bone destruction and its pattern, zone of transition, type of periosteal reaction, type of matrix calcification, skip metastasis, and soft tissue mass. The data analyzed with reliability testing through agreement analysis and Kappa value.Results: After evaluation of 30 radiographs of osteosarcoma, substantial reliability score for periosteal reaction (Kappa 0.79) to almost perfect for bone destruction, types of bone destruction, transitional zone, types of periosteal reaction, calcification matrix, and presence of soft tissue mass (Kappa 0.84-1) were found. There is a strong level of agreement (96.7%) found for skip metastasis. Intraobserver reliability also showed substantial value for the presence of soft tissue mass (Kappa 0.78) and almost perfect for other characteristics of osteosarcoma (0.84-0.93).Conclusion: Intraobserver and interobserver reliability of osteosarcoma radiograph interpretation was substantial and almost perfect.

Angiographic results of surgical or endovascular treatment of intracranial aneurysms: a systematic review and inter-observer reliability study.

Results of surgical or endovascular treatment of intracranial aneurysms are often assessed using angiography. A reliable method to report results irrespective of treatment modality is needed to enable comparisons. Our goals were to systematically review existing classification systems, and to propose a 3-point classification applicable to both treatments and assess its reliability. We conducted two systematic reviews on classification systems of angiographic results after clipping or coiling to select a simple 3-category scale that could apply to both treatments. We then circulated an electronic portfolio of angiograms of clipped (n=30) or coiled (n=30) aneurysms, and asked raters to evaluate the degree of occlusion using this scale. Raters were also asked to choose an appropriate follow-up management for each patient based on the degree of occlusion. Agreement was assessed using Krippendorff's α statistics (αK), and relationship between occlusion grade and clinical management was analyzed using Fisher's exact and Cramer's V tests. The systematic reviews found 70 different grading scales with heterogeneous reliability (kappa values from 0.12 to 1.00). The 60-patient portfolio was independently evaluated by 19 raters of diverse backgrounds (neurosurgery, radiology, and neurology) and experience. There was substantial agreement (αK=0.76, 95%CI, 0.67-0.83) between raters, regardless of background, experience, or treatment used. Intra-rater agreement ranged from moderate to almost perfect. A strong relationship was found between angiographic grades and management decisions (Cramer's V: 0.80±0.12). A simple 3-point scale demonstrated sufficient reliability to be used in reporting aneurysm treatments or in evaluating treatment results in comparative randomized trials.

Ultrasound measurement of magnetically controlled growing rod (MAGEC) lengthening. An intra and interobserver reliability study

Ultrasound measurement of magnetically controlled growing rod (MAGEC) lengthening. An intra and interobserver reliability study

Feasibility and reliability of a German guideline for farm's self-monitoring in sows and piglets.

The present study's aim was to test a German guideline for farm's self-monitoring in sows and piglets for its feasibility as well as its interobserver and test-retest reliability. The study was performed between September 2016 and April 2018 on 13 farrowing farms in Northern Germany. Contrary to the guideline, the testing was not carried out by the farmers themselves but by 2 observers with experience in pigs representing common farmers. For the interobserver reliability study, the observers performed 20 joint farm visits independently assessing the same animals. For the test-retest reliability study, each farm was visited 5 times by 1 observer (day 0, day 3, week 7, month 5, month 10). Farm visit 1 (day 0) was used as reference and compared with the remaining farm visits. The reliability was evaluated using Spearman's rank correlation coefficient (RS), intraclass correlation coefficient (ICC), and limits of agreement (LoA). As results, the guideline's feasibility was limited. The indicators' reliability also presented divergent results: All indicators in piglets revealed acceptable to good interobserver reliability (RS 0.64 to 0.77 ICC 0.33 to 0.48 LoA between the intervals -0.02 to 0.13 and -0.02 to 0.00). Contrarily, interobserver reliability was low for indicators in sows (e.g., claw alterations: RS -0.41 ICC 0.00 LoA -0.97 to 0.68). Overall acceptable test-retest reliability could be assigned to all indicators although no exact agreement existed but only the pursuit of trends is indicated. On the basis of the present results, the guideline in its present form cannot be recommended for farms' self-monitoring. Given the restricted applicability, it should be reconsidered whether it might not be more appropriate to use scientifically tested and generally accepted animal welfare assessment systems such as the Welfare Quality animal welfare assessment protocols, although their application is also time-consuming.

The Effect of Clinical Knowledge and Surgical Experience on Treatment Choice in Legg-Calve-Perthes Disease: Intra-observer and Inter-observer Reliability Study.

The aim of this study was to evaluate standardized hip radiographs, arthographs, demographic characteristics, physical examination findings, and their effects on treatment choices in leg-calve-perthes disease (LCPD). Intraobserver and interobserver realibility between orthopaedic residents, orthopaedic surgeons, and paediatric orthopaedic surgeons were also investigated. 47 LCPD patients were included this cross-sectional study. Six separate presentations including different variabilities (clinical findings, standard radiographs, and arthrographs) were evaluated by three different groups (residents, surgeons, paediatric orthopaedic surgeons) and were sent to the observers every other month by hiding patients' personal information. Seven different treatment modalities were introduced for the best treatment modality. Intraobserver and interobserver reliability in these three groups were examined. Percentage aggreement (PA) and intraclass correlation coefficients (ICC) tests were used for this purpose. Treatment PA rates between presentations were 29.5-53.6% in residents, 38.3-60.4% in surgeons, and 39.1-59.8% in pediatric orthopaedic surgeons. Conservative methods were mostly preferred as treatment modality in all groups; followed by proximal femoral osteotomies. Pediatric orthopaedic surgeons preferred safe dislocation and femur head and/or neck reconstruction surgery 5-18 times more than residents and orthopaedic surgeons. Intraobserver reliability of treatment modalities was higher among the paediatric orthopaedic surgeons; the results were fair-good (0.483-0.763). Among residents and orthopaedic surgeons, ICC values were poor to good, respectively (- 0.080 to 0.636 and 0.263-0.643). Interobserver reliability among three groups was meanly good. As surgical experience increases, both the compliance rates of the treatment modalities and the intra- and inter-group reliability are increased. Knowledge of the demographic data and clinical findings of patients besides hip radiographs or arthrographs increase treatment compliance in paediatric orthopaedic surgeons, however, cause changes in treatment modalities in residents. As surgical experience increases, more difficult surgeries such as safe dislocation and femoral head/neck reconstruction are preferred.

The Comprehensive AO CMF Classification System for Mandibular Fractures: A Multicenter Validation Study.

The AO CMF has recently launched the first comprehensive classification system for craniomaxillofacial (CMF) fractures. The AO CMF classification system uses a hierarchical framework with three levels of growing complexity (levels 1, 2, and 3). Level 1 of the system identifies the presence of fractures in four anatomic areas (mandible, midface, skull base, and cranial vault). Level 2 variables describe the location of the fractures within those defined areas. Level 3 variables describe details of fracture morphology such as fragmentation, displacement, and dislocation. This multiplanar radiographic image-based AO CMF trauma classification system is constantly evolving and beginning to enter worldwide application. A validation of the system is mandatory prior to a reliable communication and data processing in clinical and research environments. This interobserver reliability and accuracy study is aiming to validate the three current modules of the AO CMF classification system for mandible trauma in adults. To assess the performance of the system at the different precision levels, it focuses on the fracture location within the mandibular regions and condylar process subregions as core components giving only secondary attention to morphologic variables. A total of 15 subjects individually assigned the location and features of mandibular fractures in 200 CT scans using the AO CMF classification system. The results of these ratings were then statistically evaluated for interobserver reliability by Fleiss' kappa and accuracy by percentage agreement with an experienced reference assessor. The scores were used to determine if the variables of levels 2 and 3 were appropriate tools for valid classification. Interobserver reliability and accuracy were compared by hierarchy of variables (level 2 vs. level 3), by anatomical region and subregion, and by assessor experience level using Kruskal-Wallis and Wilcoxon's rank-sum tests. The AO CMF classification system was determined to be reliable and accurate for classifying mandibular fractures for most levels 2 and 3 variables. Level 2 variables had significantly higher interobserver reliability than level 3 variables (median kappa: 0.69 vs. 0.59, p < 0.001) as well as higher accuracy (median agreement: 94 vs. 91%, p < 0.001). Accuracy was adequate for most variables, but lower reliability was observed for condylar head fractures, fragmentation of condylar neck fractures, displacement types and direction of the condylar process overall, as well as the condylar neck and base fractures. Assessors with more clinical experience demonstrated higher reliability (median kappa high experience 0.66 vs. medium 0.59 vs. low 0.48, p < 0.001). Assessors with experience using the classification software also had higher reliability than their less experienced counterparts (median kappa: 0.76 vs. 0.57, p < 0.001). At present, the AO CMF classification system for mandibular fractures is suited for both clinical and research settings for level 2 variables. Accuracy and reliability decrease for level 3 variables specifically concerning fractures and displacement of condylar process fractures. This will require further investigation into why these fractures were characterized unreliably, which would guide modifications of the system and future instructions for its usage.

Tumor budding in colorectal carcinoma: An institutional interobserver reliability and prognostic study of colorectal adenocarcinoma cases

BackgroundColorectal carcinomas are one of the most commonly diagnosed malignancies. There are many prognostic factors relating to clinical course and disease progression, including tumor stage, metastasis, and tumor budding. In 2016, the International Tumor Budding Consensus Conference (ITBCC) created a system to uniformly assess tumor budding. This system includes a 3-tier system for the grading of tumor budding. In the past, there lacked uniform consensus, however the general grading practice was based on a 2-tiered system. Given that tumor budding is considered to have prognostic value, the accuracy and reproducibility of its assessment is vital. Our study aims to look at interobserver agreement in the scoring of tumor budding. DesignA total of 233 cases of colorectal carcinoma diagnosed in our health system were retrospectively analyzed and routine H&E stained slides of these cases were collected. A representative slide for tumor budding was selected per case. Four investigators with different levels of experience and expertise evaluated the selected slide of each case for tumor budding. Scoring was based on the ITBCC protocol. Clinico-pathological data was collected for each case and analyzed with tumor budding scores. Tumor budding scores per individual investigator and consensus tumor budding score were compared to patient and tumor characteristics including patient survival, tumor grade, tumor stage, and lymph node status. ResultsInter-observer agreement was calculated using Gwet's Agreement Coefficient (AC1) and associated 95% confidence intervals was used to compare the ratings made by 4 pathologists. Overall, there was variation among pathologists in tumor budding score (Gwet's agreement coefficient = 0.25 and 0.326 for 3-tier and 2-tier grading system, respectively). Results show higher reliability with the 2-tier system compared to the 3-tier system. Tumor stage was significantly associated with budding score for all individual investigators and the consensus value (p value < 0.001). ConclusionThere is low inter-observer agreement in the assessment of tumor budding in colorectal carcinoma. This suggests that it is difficult to uniformly grade tumor budding and that our classification system needs improvement. We found that the older 2-tier system (Hase et al.) results in slightly higher inter-observer agreement than the recently proposed 3-tier grading system (ITBCC, 2016), though both systems lead to suboptimal agreement. Worth noting is that observers with subspecialty GI training and more work experience had higher inter-observer agreement. Our results showed that subspecialty training tends to increase agreement more than overall work experience. In addition, our exploratory results showed that there is an association of tumor budding score to tumor stage. While increasing refinement in classification, the 3-tiered system resulted in decreased agreement in tumor budding assessment. Clearly, there is more work to be done in the identification and quantification of tumor buds.

Nodular Leptomeningeal Disease—A Distinct Pattern of Recurrence After Postresection Stereotactic Radiosurgery for Brain Metastases: A Multi-institutional Study of Interobserver Reliability

For brain metastases, surgical resection with postoperative stereotactic radiosurgery is an emerging standard of care. Postoperative cavity stereotactic radiosurgery is associated with a specific, underrecognized pattern of intracranial recurrence, herein termed nodular leptomeningeal disease (nLMD), which is distinct from classical leptomeningeal disease. We hypothesized that there is poor consensus regarding the definition of LMD, and that a formal, self-guided training module will improve interrater reliability (IRR) and validity in diagnosing LMD. Twenty-two physicians at 16 institutions, including 15 physicians with central nervous system expertise, completed a 2-phase survey that included magnetic resonance imaging and treatment information for 30 patients. In the "pretraining" phase, physicians labeled cases using 3 patterns of recurrence commonly reported in prospective studies: local recurrence (LR), distant parenchymal recurrence (DR), and LMD. After a self-directed training module, participating physicians completed the "posttraining" phase and relabeled the 30 cases using the 4 following labels: LR, DR, classical leptomeningeal disease, and nLMD. IRR increased 34% after training (Fleiss' Kappa K = 0.41 to K = 0.55, P < .001). IRR increased most among non-central nervous system specialists (+58%, P < .001). Before training, IRR was lowest for LMD (K = 0.33). After training, IRR increased across all recurrence subgroups and increased most for LMD (+67%). After training, ≥27% of cases initially labeled LR or DR were later recognized as nLMD. This study highlights the large degree of inconsistency among clinicians in recognizing nLMD. Our findings demonstrate that a brief self-guided training module distinguishing nLMD can significantly improve IRR across all patterns of recurrence, and particularly in nLMD. To optimize outcomes reporting, prospective trials in brain metastases should incorporate central imaging review and investigator training.