Intraobserver Disagreement Research Articles

Deep learning approach to detect high-risk oral epithelial dysplasia: A step towards computer-assisted dysplasia grading

Introduction: Oral epithelial dysplasia (OED) is associated with high interobserver and intraobserver disagreement. With the exponential increase in the applicability of artificial intelligence tools such as deep learning (DL) in pathology, it would now be possible to achieve high accuracy and objectivity in grading of OED. In this research work, we have proposed a DL approach to epithelial dysplasia grading by creating a convolutional neural network (CNN) model from scratch. Materials and Methods: The dataset includes 445 high-resolution ×40 photomicrographs captured from histopathologically diagnosed cases of high-risk dysplasia (HR) and normal buccal mucosa (NBM) that were used to train, validate and test the two-dimensional CNN (2DCNN) model. Results: The whole dataset was divided into 60% training set, 20% validation set and 20% test set. The model achieved training accuracy of 97.21%, validation accuracy of 90% and test accuracy of 91.30%. Conclusion: The DL model was able to distinguish between normal epithelium and HR epithelial dysplasia with high grades of accuracy. These results are encouraging for researchers to formulate DL models to grade and classify OED using various grading systems.

POS0036 AN ARTIFICIAL INTELLIGENCE MODEL IN RHEUMATOLOGY: INTERPRETATION OF THE SACROILIAC JOINT GRAPHY IN ANKYLOSING SPONDYLITIS

Background:There is an average of 8 years delay in the diagnosis of ankylosing spondylitis (AS). The most important danger of late diagnosis is that the disease can cause physical and functional disability (2). There is no specific diagnostic biomarker for AS. Sacroiliac joint (SIJ) radiography is frequently used in the diagnosis and follow-up of AS due to its easy accessibility and low cost. It can be classified as grade 0, 1, 2, 3, 4, and these classes may not be sharply separated from each other (3).Objectives:Interpretation of the SIJ radiography may differ from physician to physician. In fact, the same physician may interpret it differently at different times (3). We wanted to find a solution to the intraobserver disagreement problem with the artificial intelligence model.Methods:The SIJ radiography of 590 patients who applied to our center were divided into 3 categories as right and left, separately, grade 0, grade 1-2, grade 3-4, and an educational data set was prepared for the object recognition method. 488 images were augmented through noise from 490 images in the training data. 242 articular objects were trained for grade 0, 278 for grade 1-2, and 1426 for grade 2-3. The model was tested with 100 images for 36 joint objects for grade 0, 29 for grade 1-2, and 135 for grade 3-4 to create a computer vision-artificial intelligence model (image 1).Results:Training performance is 70% for grade 0, %63 for grade 1-2, %90 for grade 3-4 and test performance is %52 for grade 0, %24 for grade 1-2, %86 intersection over union (I/U:Intersection over Union is a form of measurement used to indicate the accuracy of an object detector.) for grade 3-4. The mean average precision (mAP) score of our object detection model is %65.9 for test data set (image 1). The estimation quality of the model can be affected by the distribution and number of each class.Conclusion:The experience of the x-ray technician, dose adjustment, and position differences due to patient compliance complicate the standardization of SIJ radiography and this may cause interobserver disagreement (3). Artificial intelligence models to be created with a larger and homogeneous data set in order to ensure objective standardization in the interpretation of the SIJ graph can help physicians.

Clustering with Pairs of Prototypes to Support Automated Assessment of the Fetal State

ABSTRACTCardiotocographic (CTG) monitoring, consisting in analysis of the fetal heart rate, uterine contractions, and fetal movements is the primary noninvasive method for the fetal state assessment. The visual interpretation of the CTG signals is characterized by the large inter- and intraobserver disagreement. Hence, the automated methods supporting the diagnosis process are the topic of researches. In the presented study, the evaluation of the CTG signals, based on fuzzy clustering with pairs of prototypes, is described. The efficiency of the proposed method is verified using two benchmark datasets of the CTG signals (CTU-UHB and SisPorto), and the problems of the two- and three-class classification are considered. The obtained results show the improved quality of the automated fetal state assessment in accordance with the applied reference procedures: the fuzzy (c+p)-means clustering and the Lagrangian Support Vector Machines.

Intra-observer reproducibility of whole slide imaging for the primary diagnosis of breast needle biopsies

Background: Automated whole slide imaging (WSI), also known as virtual microscopy is rapidly becoming an important tool in diagnostic pathology. Currently, the primary utilization of the technique is for transmission of digital images, for second opinion consultation, as well as for quality assurance and education. The high-resolution of digital images along with the refinement of technology could now allow for WSI to be used as an alternative to conventional microscopy (CM) as a first line diagnostic platform. However, the accuracy and reproducibility of the technology for the routine histopathologic diagnosis has not been established yet. This study was undertaken to compare the intra-observer variability of WSI and CM in the primary diagnosis of breast biopsies. Materials and Methods: One hundred and three consecutive core needle biopsies of breast were selected for this study. Each slide was digitally scanned and the images were stored in a shared file. Three board-certified pathologists independently reviewed the glass slides by CM first, and in an interval of 2-3 weeks for the 2nd time to establish their baseline CM versus CM reproducibility. They then reviewed the digital images of all cases following the same interval of time to compare the reproducibility of WSI versus CM for each observer. The diagnostic categories included the typical range of benign and malignant mammary lesions. Results: The intra-observer variability for CM versus CM was 4%, 7%, and 0% for observers 1, 2, and 3 respectively. The diagnostic variability for WSI versus CM was 1%, 4%, and 1% for the same observers. All diagnostic disagreements were between ductal hyperplasia and atypical ductal hyperplasia. There was no intra-observer disagreement in the diagnosis of benign versus malignant disease. Conclusions: The intra-observer variability in the diagnosis of the core needle biopsies of the breast by high-resolution, WSI was the same as conventional glass slide microscopy. These results suggest that, WSI could be used similar to CM for the initial diagnosis of breast biopsies.

A method to analyse observer disagreement in visual grading studies: example of assessed image quality in paediatric cerebral multidetector CT images

The purpose was to demonstrate a non-parametric statistical method that can identify and explain the components of observer disagreement in terms of systematic disagreement as well as additional individual variability, in visual grading studies. As an example, the method was applied to a study where the effect of reduced tube current on diagnostic image quality in paediatric cerebral multidetector CT (MDCT) images was investigated. Quantum noise, representing dose reductions equivalent to steps of 20 mA, was artificially added to the raw data of 25 retrospectively selected paediatric cerebral MDCT examinations. Three radiologists, blindly and randomly, assessed the resulting images from two different levels of the brain with regard to the reproduction of high- and low-contrast structures and overall image quality. Images from three patients were assessed twice for the analysis of intra-observer disagreement. The intra-observer disagreement in test-retest assessments could mainly be explained by a systematic change towards lower image quality the second time the image was reviewed. The inter-observer comparisons showed that the paediatric radiologist was more critical of the overall image quality, while the neuroradiologists were more critical of the reproduction of the basal ganglia. Differences between the radiologists regarding the extent to which they used the whole classification scale were also found. The statistical method used was able to identify and separately measure a presence of bias apart from additional individual variability within and between the radiologists which is, at the time of writing, not attainable by any other statistical approach suitable for paired, ordinal data.

Colonoscopy Peer Review Utilizing Automated Video Capture

Purpose: Colonoscopy is the screening modality of choice for detecting early colorectal cancer in the United States. Several retrospective reports show that colorectal cancers may appear within 3 years of a colonoscopy, suggesting some polyps or cancers are missed. Despite these findings, no standardized, validated method exists to objectively evaluate the skill set of an endoscopist. Aim: to evaluate an anonymous, automated video capture system used in conjunction with novel software to facilitate peer review of endoscopic performance, and to see if higher quality exams correlate with withdrawal time. Methods: Existing infrastructure and software were used to capture entire colonoscopies which were then stored as digital video files and reviewed anonymously. Video colonoscopy files were randomly obtained from 2 endoscopy suites at Mayo Clinic-Rochester over the course of 4 days in 2007. Three endoscopists with varied endoscopic experience (1 fellow, 1 junior staff and 1 senior staff) reviewed a set of 24 anonymous video colonoscopies, and provided feedback via a graphical user interface. The 3 reviewers were asked to assign numerical scores on several parameters: adequacy of colonic mucosal visualization including the proximal side of haustral folds, flexures, rectal valves and the ileocecal valve (0-100), adequacy of removal of fluid/feces during exam (0-10), and adequacy of luminal distention during exam (0-10). Additionally, reviewers identified key time points during each colonoscopy and a withdrawal time was calculated. Results: The panel of 3 individual reviewers demonstrated agreement when asked to evaluate 24 random colonoscopy video files of varied quality. Specifically, the intraobserver agreement for the visualization score (0-100) was good, reflected by a disagreement range of 19 ± 13 (Mean+SD). Similarly, scores for adequacy of fluid/feces removal (0-10) and adequacy of luminal distention (0-10), showed good agreement also reflected by low intra-observer disagreement; range 2.0 ± 1.3, and 2.1 ± 1.5 (Mean+SD), respectively. There was no relationship between visualization score and withdrawal time for each reviewed case: least squares analysis showed no correlation between the two (R2 = 0.04). Conclusion: Our results show that a standardized, anonymous method utilizing video capture and new software technology allows effective and consistent peer review of individual colonoscopies. The close agreement among reviewers suggests that this peer review method is independent of the evaluator's experience. Therefore, our method can be used to ensure that an endoscopist meets a minimal standard of quality, assessed in an anonymous and reproducible fashion. Our results question the utility of withdrawal time as a standard for quality. Disclosure: PC de Groen - stock in EndoMetric J Wong - stock in EndoMetric W Tavanapong - stock in EndoMetric J Oh - stock in EndoMetric.

Three‐dimensional endoanal ultrasonography: intraobserver and interobserver agreement using scoring systems for classification of anal sphincter defects

To determine the degree of intraobserver and interobserver agreement for an experienced and an inexperienced sonologist using two scoring systems for ultrasonographic assessment of anal sphincter defects. The study sample comprised the datasets of all women aged between 20 and 40 years who had attended our outpatient clinic and undergone a complete three-dimensional (3D) endoanal ultrasound (EAUS) examination in the period from January 2003 to December 2005. The EAUS datasets were assessed twice independently by two sonologists: one with experience of > 400 3D EAUS assessments and one inexperienced sonologist who had performed approximately 50 assessments before the study. Cases with intraobserver disagreement were resolved by a third (final) assessment. The final assessment from each observer was used to determine the degree of interobserver agreement. Sphincter defects were classified according to our EAUS defect score and the Starck score. EAUS datasets of 55 women were included. Based on first vs. second assessments, intraobserver agreement for the experienced sonologist was good for our EAUS defect score (weighted kappa, 0.75) and the Starck score (weighted kappa, 0.73). Intraobserver agreement for the inexperienced sonologist was moderate for our EAUS defect score (weighted kappa, 0.58) and good for the Starck score (weighted kappa, 0.62). Interobserver agreement was good for both our EAUS defect score (weighted kappa, 0.65) and the Starck score (weighted kappa, 0.74). Intraobserver and interobserver agreement was acceptable for both scoring systems. The experienced sonologist obtained a higher degree of intraobserver agreement than did the inexperienced sonologist.

Cardiac MRI: Standardized Right and Left Ventricular Quantification by Briefly Coaching Inexperienced Personnel

MRI quantification of the cardiac ventricles is time-consuming, especially for the right ventricle due to its complex geometry. Using a short axis MRI protocol, we tested if briefly coaching persons inexperienced in cardiac MRI provides reliable right and left ventricular quantification. 22 healthy subjects (mean age 26 ± 4.2 years) underwent short-axis breath-hold SSFP sequence cardiac MRI. Two per- sons inexperienced in cardiac MRI independently traced endocardial and epicardial contours of both ventricles with a pre- defined contour tracing protocol. Measurements were repeated after visual correction of each two most basal slices of the endocardial contours and epicardial contours in the learning curve. Five random short-axis cines were retraced for intra- observer variability. Measurements were performed blinded within 4 months. Agreement was assessed with the Bland- Altman method. No systematic bias was observed and measurements were within acceptable limits of agreement after brief coaching. Re- peat measurements following visual correction significantly improved inter-observer differences, especially for mass cal- culations. Maximum interobserver and intraobserver disagreement of the final protocol were respectively � 8% and � 5%. Brief coaching of persons inexperienced in cardiac MRI, using the short axis MRI protocol, provides reliable volume, function and mass quantification of both ventricles.

Comparative localized linear accuracy of small-field cone-beam CT and multislice CT for alveolar bone measurements

To compare the accuracy of cone-beam computerized tomography (CBCT) and multislice CT (MSCT) for linear jaw bone measurements. An ex vivo formalin-fixed human maxilla was imaged with both CBCT (Accuitomo 3D; Morita, Kyoto, Japan) and MSCT (4-slice Somatom VolumeZoom and 16-slice Somatom Sensation 16; Siemens, Erlangen, Germany). The MSCT images were reconstructed using different reconstruction filters to optimize bone visualization (U70u and U90u for VolumeZoom, H30s and H60s for Sensation 16). Before scanning, triplets of small gutta-percha markers were glued onto the soft tissues overlying the maxillary bone on the top and on both sides of the alveolar ridge to define a set of reproducible linear measurements in 11 planes. Image measurements were performed by 2 observers. The gold standard was determined by means of physical measurements with a caliper by 3 observers. The accuracy of the linear measurements was 0.35 +/- 1.31 mm (U70u) and 0.06 +/- 1.23 mm (U90u) for the Somatom VolumeZoom, 0.24 +/- 1.20 mm (H60s) and 0.54 +/- 1.14 mm (H30s) for the Sensation 16, and -0.09 +/- 1.64 mm for the Accuitomo 3D. Statistical analysis with 2-way analysis of variance showed no significant inter- or intraobserver disagreement for the physical or the radiologic measurements. There was also no significant difference for the measurements on the different reconstruction filters. Both CBCT and MSCT yield submillimeter accuracy for linear measurements on an ex vivo specimen.

Inter‐ and intraobserver reproducibility of thyroid fine needle aspiration cytology: an analysis of discrepant cases

To determine the inter- and intraobserver reproducibility and analyse the discrepant cases of fine needle aspiration cytology (FNAC) of the thyroid. Cases of thyroid FNAC with a corresponding histological diagnosis were reviewed regarding the original cytological diagnoses by two observers. The final cytological diagnoses (FCD) included both concordant and consensus diagnoses. The inter- and intraobserver reproducibility and efficacy of thyroid FNAC were calculated based on the FCD. A total of 97 FNAC cases with corresponding histopathological specimens were analysed. Although inter- and intraobserver disagreement in the cytological diagnoses occurred in about one-quarter of the cases analysed (24.7% and 23.7%, respectively), a substantial level of diagnostic interobserver (kappa = 0.71) and intraobserver (kappa = 0.66) reproducibility was observed. The efficacy of the method was 94.4%. Disagreement in the diagnosis was detected in 24 cases (24.7%), most of them (41.7%) for follicular lesions. Discordant cytological diagnoses between the two observers were represented by six (16.2%) of the 37 cases with an FCD of colloid nodule, five (41.7%) of the 12 cases of cellular follicular lesion, all three cases of follicular neoplasm, in two (6.3%) of the 32 cases of PTC, one (16.7%) of six cases of follicular neoplasm with a predominance of Hürthle cells and in one case of poorly differentiated neoplasia. Similarly, major disagreement in intraobserver cytological diagnoses was observed for the diagnosis of follicular lesions: 18 (78.3%) of a total of 23 discordant cases. As discrepancies in the cytopathological diagnosis can have repercussions in the management of patients, all cases with a cytological diagnosis of follicular lesions/neoplams should be reviewed in multidisciplinary meetings thus minimizing interobserver variability.

How should we grade CIN?

How should we grade CIN? The grading of cervical intraepithelial neoplasia (CIN) is problematic. CIN represents a morphological continuum, but biopsies displaying this lesion are classified into two (e.g. Bethesda) or three grade categories, sometimes with poor reproducibility. There are also difficulties in reliably distinguishing low-grade CIN from its reactive simulants. Because of problems with inter- and intra-observer disagreement in the grading of CIN and the diagnosis of low-grade lesions, three expert contributions were commissioned to address the question 'how should we grade CIN?'.

Flow cytometric DNA analysis and p53 protein expression show a good correlation with histologic findings in patients with Barrett's esophagus.

There is a considerable degree of subjectivity and, therefore, substantial interobserver and intraobserver disagreement in the diagnosis and grading of dysplastic lesions in Barrett's esophagus (BE). The aim of this study was to evaluate the usefulness of DNA flow cytometry and immunohistochemical staining for p53 protein as objective methods to complement the conventional histologic diagnosis of dysplasia in patients with this disease. The most common problems and the possible advantages of using these procedures are analyzed briefly in this article. Formalin fixed, paraffin embedded tissue from 55 patients diagnosed with BE were processed for flow cytometric measurements (ploidy and proliferation index) and p53 immunostaining. Both the cytometric data and the positivity of staining for p53 revealed a statistically significant increase throughout the following sequence: no dysplasia --> indefinite for dysplasia --> low grade dysplasia --> high grade dysplasia --> adenocarcinoma. There was also a highly significant correlation between the results of the cytometric study and the positivity of staining for p53. In the future, the use of this procedure could play an important role in the evaluation of patients with BE. Considering that staining for p53 is technically simple, economical, and quick, and the materials required are available to most pathology laboratories, this method appears to be a firm candidate for application as a biomarker in BE. The authors have shown that it is possible to obtain adequate results for cytometric analysis with small formalin fixed, paraffin embedded biopsies if a strict protocol for the acceptance of tissue samples and/or histograms is observed.

Ventilation-Perfusion Lung Scanning and the Diagnosis of Pulmonary Embolism: Improvement of Observer Agreement by the Use of a Lung Segment Reference Chart

To test the hypothesis that the systematic use of a lung segment reference chart can improve the inter- and intra-observer agreement for the interpretation of ventilation-perfusion lung scans. A randomized trial. Ventilation-perfusion lung scans were obtained in a series of 220 consecutive patients with clinically suspected pulmonary embolism. Ventilation-perfusion scans were randomly allocated to one of two series each consisting of 110 ventilation-perfusion lung scans. The first series of lung scans was interpreted according to the routine diagnostic approach, and the second series was interpreted with the mandatory use of a lung segment reference chart on which observed ventilation and perfusion defects were drawn. The two nuclear medicine physicians agreed a priori on the diagnostic criteria of the classification scheme. Lung scans were classified as normal, non-high probability, or high probability for pulmonary embolism. The extent of disagreement between the nuclear medicine physicians (inter-observer disagreement) and the lack of internal consistency of each nuclear medicine physician (intra-observer disagreement) was assessed by the percentage disagreement and by kappa statistic. Inter-observer disagreement which was 20% in the first series, decreased significantly in the second series to 7%; P = 0.003. Intra-observer disagreement for the first series was 10% and 22% for the nuclear medicine physicians, respectively. Intra-observer disagreement for the second series of lung scans decreased significantly for one nuclear medicine physician (intra-observer disagreement, 0%; P less than 0.01), whereas intra-observer disagreement was reduced to 10% for the other nuclear medicine physician (P = 0.09). Inter- and intra-observer disagreement were significantly reduced when two nuclear medicine specialists interpreted ventilation-perfusion lung scans according to the routine diagnostic approach plus the use of a lung segment reference chart. The use of the lung segment reference chart for the interpretation of lung scans is likely to improve the management of patients with clinically suspected pulmonary embolism.

Histologic grading of prostatic adenocarcinoma: Intraobserver reproducibility of the Mostofi, Gleason and Böcking grading systems

Intraobserver variation of three grading systems--Mostofi, Gleason and Böcking--is examined. No significant difference was noted between the histological grades found in the two examinations by any of the three methods used. Neither the type of surgical procedure nor the number of slices with tumour influenced the reproducibility of histological grading within each system studied. In the Gleason system the intraobserver highest disagreement would not have resulted in change of therapy choice, but in 2% of tumours graded according to the Mostofi system this would have occurred if the choice of therapy would depend on the grading results.

A new method of estimation of interobserver variation and its application to the radiological assessment of osteoarthrosis in hip joints.

Four frequently referenced radiological methods for assessment of osteoarthrosis in hip joints are compared with respect to their interobserver variation. One hundred hip joints were rated by four observers. Each observer applied all four methods to all the radiographs. The results were analysed by a new statistical method, which allowed detailed assessment of the nature and extent of interobserver variation. Significant differences in terms of intraobserver disagreement were found, and Heripret's method was the best in this respect. All four methods showed a significant degree of systematic disagreement among the observers. This suggests that the interobserver variation might be reduced by confronting the observers with the results of the statistical analysis of their judgements.

Temporal Arteritis: A Clinicopathologic Study

Clinical and laboratory findings of 237 consecutive patients who had 250 temporal artery biopsies performed at the Wilmer Institute during a 15-year period were reviewed. Biopsies were reviewed independently on three separate occasions by four observers who, unaware of the clinical history, made one of the following histopathologic diagnoses: active arteritis, healed arteritis, arteriosclerosis, atherosclerosis, normal, or other. Biopsies were reviewed a fourth time by the observers together to establish a consensus diagnosis. The frequency of intraobserver disagreement in biopsy interpretation ranged from 4.4 to 25.6% of cases. The frequency with which observers disagreed with the consensus diagnosis (interobserver variation) ranged from 4.3 to 13.5% of cases. Pre-biopsy steroid therapy for seven to eight weeks or longer was associated with loss of the histopathologic features of active arteritis. We recommend biopsies for all patients with known contraindications to steroid therapy. In addition, multivariant regression analysis suggests that certain patients can be selected who are likely to have positive biopsies. For some patients, the clinical diagnosis of temporal arteritis can be based on the clinical signs and symptoms and their response to a therapeutic trial of steroids.

Detection accuracy in chest radiography.

The detection accuracy of the diagnostic radiologist is important in everyday medical decision making. However, little work has been done relating the detection accuracy of the radiologist to the quality of the image. This study, using a thorax and lung phantom, simulated tissue-equivalent 6.4 mm lesions, and a 183 cm source-to-image distance, shows that the detection accuracy is not dependent on the focal spot size (over a range of 0.3-2.0 mm). However, the false positive rate increases when using small focal spots. In addition, the detection accuracy decreases with increasing root-mean-square (RMS) noise (a measure of the amount of quantum mottle in the image), while the false positive rate and intraobserver disagreement increase with increasing RMS noise. It is also shown that the nonradiologist responds to changes in noise in exactly the same way as the radiologist.