Impact Of Interobserver Variability Research Articles

Reproducibility and prognostic ability of chronicity parameters in kidney biopsy – Comprehensive evaluation comparing microscopy and artificial intelligence in digital pathology

IntroductionSemi-quantitative scoring of various parameters in renal biopsy is accepted as an important tool to assess disease activity and prognostication. There are concerns on the impact of interobserver variability in its prognostic utility, generating a need for computerized quantification. MethodsWe studied 94 patients with renal biopsies, 45 with native diseases and 49 transplant patients with index biopsies for Polyomavirus nephropathy. Chronicity scores were evaluated using two methods. A standard definition diagram was agreed after international consultation and four renal pathologists scored each parameter in a double-blinded manner. Interstitial fibrosis (IF) score was assessed with five different computerized and AI-based algorithms on trichrome and PAS stains. ResultsThere was strong prognostic correlation with renal function and graft outcome at a median follow-up ranging from 24 to 42 months respectively, independent of moderate concordance for pathologists scores. IF scores with two of the computerized algorithms showed significant correlation with estimated glomerular filtration rate (eGFR) at biopsy but not at the end of follow-up. There was poor concordance for AI based platforms. ConclusionChronicity scores are robust prognostic tools despite interobserver reproducibility. AI-algorithms have absolute precision but are limited by significant variation when different hardware and software algorithms are used for quantification.

Impact of Interobserver Variability in Manual Segmentation of Non-Small Cell Lung Cancer (NSCLC) Applying Low-Rank Radiomic Representation on Computed Tomography.

Simple SummaryDiscovery of predictive and prognostic radiomic features in cancer is currently of great interest to the radiologic and oncologic community. Tumor phenotypic and prognostic information can be obtained by extracting features on tumor segmentations, and it is typically imaging analysts, physician trainees, and attending physicians who provide these labeled datasets for analysis. The potential impact of level and type of specialty training on interobserver variability in manual segmentation of NSCLC was examined. Although there was some variability in segmentation between readers, the subsequently extracted radiomic features were overall well correlated. High fidelity radiomic feature extraction relies on accurate feature extraction from imaging that produce robust prognostic and predictive radiomic NSCLC biomarkers. This study concludes that this goal can be obtained using segmenters of different levels of training and clinical experience.This study tackles interobserver variability with respect to specialty training in manual segmentation of non-small cell lung cancer (NSCLC). Four readers included for segmentation are: a data scientist (BY), a medical student (LS), a radiology trainee (MH), and a specialty-trained radiologist (SK) for a total of 293 patients from two publicly available databases. Sørensen–Dice (SD) coefficients and low rank Pearson correlation coefficients (CC) of 429 radiomics were calculated to assess interobserver variability. Cox proportional hazard (CPH) models and Kaplan-Meier (KM) curves of overall survival (OS) prediction for each dataset were also generated. SD and CC for segmentations demonstrated high similarities, yielding, SD: 0.79 and CC: 0.92 (BY-SK), SD: 0.81 and CC: 0.83 (LS-SK), and SD: 0.84 and CC: 0.91 (MH-SK) in average for both databases, respectively. OS through the maximal CPH model for the two datasets yielded c-statistics of 0.7 (95% CI) and 0.69 (95% CI), while adding radiomic and clinical variables (sex, stage/morphological status, and histology) together. KM curves also showed significant discrimination between high- and low-risk patients (p-value < 0.005). This supports that readers’ level of training and clinical experience may not significantly influence the ability to extract accurate radiomic features for NSCLC on CT. This potentially allows flexibility in the training required to produce robust prognostic imaging biomarkers for potential clinical translation.

PO-1798 Impact of inter-observer variability on MRI radiomic features in bladder cancer

PO-1798 Impact of inter-observer variability on MRI radiomic features in bladder cancer

The impact of inter-observer variability on the accuracy, precision and utility of a commonly-used grassland condition index

Inter-observer variability in ecological surveying can reduce the robustness and reliability of measures used to inform habitat management. Grassland management is important across southern Africa. The Veld Condition Index (VCI) is a technique that combines field data on the relative abundance of grasses identified during a standardised survey with knowledge of how different grass species respond to grazing. This allows calculation of an Ecological Condition Index (ECI) for a particular site. The ECI value is related to threshold values to indicate grazing pressure. Here, for the first time, the effects of inter-observer variation in trained surveyors on condition assessment and resultant management recommendations are examined. There was substantial and statistically significant inter-observer variation in ECI values, driven primarily by differences in the number of times an observer identified grasses within the sample transect as being one of the species listed in the “Decreaser” category of the standardised protocol (i.e. species that decrease with grazing). Variation was random with respect to observers rather than being due to systematic (and thus predicable and potentially correctable) differences between observers. In 44% of sites, inter-observer variation resulted in important differences in assessment of grassland condition because the assessment spanned outcome threshold categories. In such cases, this could result in unnecessary or detrimental management being recommended by some observers including inappropriate burning, needlessly translocating or culling herbivores, or failing to align herbivore populations to the carrying capacity with negative effects on both grazers and veld. We recommend that a cautious approach be taken in using VCI/ECI to inform management and that identification training be focused on species in the Decreaser ecological category that drive inter-individual variation in overall ECI.

Influence of inter-observer delineation variability on radiomics stability in different tumor sites

Background: Radiomics is a promising methodology for quantitative analysis and description of radiological images using advanced mathematics and statistics. Tumor delineation, which is still often done manually, is an essential step in radiomics, however, inter-observer variability is a well-known uncertainty in radiation oncology. This study investigated the impact of inter-observer variability (IOV) in manual tumor delineation on the reliability of radiomic features (RF).Methods: Three different tumor types (head and neck squamous cell carcinoma (HNSCC), malignant pleural mesothelioma (MPM) and non-small cell lung cancer (NSCLC)) were included. For each site, eleven individual tumors were contoured on CT scans by three experienced radiation oncologists. Dice coefficients (DC) were calculated for quantification of delineation variability. RF were calculated with an in-house developed software implementation, which comprises 1404 features: shape (n = 18), histogram (n = 17), texture (n = 137) and wavelet (n = 1232). The IOV of RF was studied using the intraclass correlation coefficient (ICC). An ICC >0.8 indicates a good reproducibility. For the stable RF, an average linkage hierarchical clustering was performed to identify classes of uncorrelated features.Results: Median DC was high for NSCLC (0.86, range 0.57–0.90) and HNSCC (0.72, 0.21–0.89), whereas it was low for MPM (0.26, 0–0.9) indicating substantial IOV. Stability rate of RF correlated with DC and depended on tumor site, showing a high stability in NSCLC (90% of total parameters), acceptable stability in HNSCC (59% of total parameters) and low stability in MPM (36% of total parameters). Shape features showed the weakest stability across all tumor types. Hierarchical clustering revealed 14 groups of correlated and stable features for NSCLC and 6 groups for both HNSCC and MPM.Conclusion: Inter-observer delineation variability has a relevant influence on radiomics analysis and is strongly influenced by tumor type. This leads to a reduced number of suitable imaging features.

EP-1646: Impact of interobserver variability and setup uncertainty on dose in organs-at-risk

EP-1646: Impact of interobserver variability and setup uncertainty on dose in organs-at-risk

Impact of Interobserver Variability in Volume Delineation on Deformable Image Registration for Prostate

Impact of Interobserver Variability in Volume Delineation on Deformable Image Registration for Prostate

Pre-implantation analysis of kidney biopsies from expanded criteria donors: testing the accuracy of frozen section technique and the adequacy of their assessment by on-call pathologists.

Pre-implantation renal biopsies of expanded criteria donors are one of the criteria used for allocation decisions, but there are concerns about the impact of the interobserver variability and the technique to be used. The aim was (i) to compare the original report performed by on-call pathologists using frozen sections (FS) to a retrospective analysis carried out by a trained pathologist using the same frozen section, and (ii) to compare the same FS to subsequently obtained paraffin sections (PS) by the same pathologist. A total of 92 biopsies, 78 from transplanted and 14 from nontransplanted cases, were analyzed. Agreement between observers using the same FS was weaker than the correlation between FS and PS in all the examined parameters (Kendall's Tau b for the Remuzzi score 0.104 vs. 0.306). According to the Remuzzi score, the revised FS analysis would have resulted in a higher rate of organ discard (n=19) than PS (n=14) and the original report (n=6). However, kidneys that would have been discarded according to the retrospective analysis showed adequate outcomes in terms of graft survival and function. Accordingly, the impact of interobserver and technique-related variability can be minimized by the use of a relatively low threshold (RS≤4) for organ acceptance.

Abstract PD1-7: Phosphatase and tensin homolog deleted from chromosome 10 (PTEN) evaluation in the NeoALTTO trial (BIG 01-06)

Abstract Introduction Assessment of PTEN might be an important tool in identifying patients unlikely to derive substantial benefit from trastuzumab and lapatinib-based therapies. Current studies failed to demonstrate an unequivocal predictive role of PTEN in human epidermal growth factor receptor-2 (HER2)-positive breast cancer both in neo-adjuvant and adjuvant settings. Lack of standardization of PTEN status determination in formalin fixed paraffin embedded (FFPE) tissue samples together with the small datasets analyzed in previous studies may have contributed to conflicting PTEN results and high variability of PTEN loss rates reported (22%-64%). Methods In the present study we have conducted a preliminary analysis investigating the influence of the antibodies, scoring methods and cutoff criteria used, together with the impact of inter-observer variability on PTEN status determination. Core biopsies from 429 FFPE baseline samples from early-stage HER2-positive breast cancer patients prospectively collected in the neoALTTO trial were stained with two widely used anti-PTEN monoclonal antibodies (clone 6H2.1 from DAKO and clone 138G6 from Cell Signaling Technology-CST). Two different pathologists independently scored the slides using the H-score system. Two different cut-off criteria were used to define PTEN loss of expression: H-score &amp;lt; 50 and ≤10% positive staining (any level of intensity). Results Out of 429 processed samples, 363 and 369 were evaluable for observer 1 (CST and DAKO, respectively) and 367 were evaluable for observer 2, with an average missing rate of 15%. A strong linear correlation was observed between CST and DAKO antibodies for both observers (R2 &amp;gt; 0.7). After setting a cutoff for PTEN loss, the observed discordance rate was below 10%, with the number of discordant cases ranging from 28 to 33 (depending on the cut-off criteria and observer). Inter-observer correlation was strong with both antibodies (R2 = 0.8). Based on the cutoff criteria and antibodies used, inter-observer discordance rates ranged between 6% and 10%; the lowest rate observed at ≤10% cutoff threshold with CST antibody. PTEN was classified as “loss” in 23%, 19%, 16%, and 13% of cases by observer 1 and in 32%, 29%, 24%, 19% of cases by observer 2 using DAKO H-score&amp;lt;50, CST H-score&amp;lt;50, DAKO≤10%, and CST≤10% criteria, respectively. Conclusion Our preliminary analysis showed high inter-antibody and inter-observer concordant rates for PTEN status determination by IHC. However, the lack of a standardized definition of PTEN cutoff thresholds for defining negativity and positivity generated loss call rates ranging from 13% to 32% within the same dataset. Correlative analyses with pCR and PIK3CA status are ongoing to determine the impact of PTEN expression and its variability in predicting response to single-agent or dual HER2 targeting. Citation Information: Cancer Res 2013;73(24 Suppl): Abstract nr PD1-7.

Impact of interobserver variability in delineating clinical target volumes (CTV) for head and neck intensity modulated radiation therapy (IMRT): Potential for a geographical miss

e17013 Background: Despite modern day imaging techniques and guidelines for delineation of the clinical target volume, there remains significant inter-observer variability in delineating the CTV. With the use of IMRT, the target volume receives a significant tumoricidal dose while the regions just outside the target receive unpredictable doses. In this report, the dose to the region just outside the planning target volume (PTV) (defined as volume of uncertainty [VOU]), presumed to represent the regions subject to maximum inter-observer variability, was studied. Methods: The IMRT plans of 12 patients with head and neck cancer were used to determine the dose just outside the high-risk CTV by growing volumes around CTV with 3 mm, 5 mm, and 7 mm margins. These volumes were edited at regions close to skin/air and bone. PTVs were subsequently grown using the same margins as used in the original plans. With the Boolean operations, each of these volumes was subtracted from the existing PTV to generate the volumes of uncertainty (VOU) in 3 dimensions. The dose to these VOUs was analyzed. D95, D90 and median dose which are the doses received by 95%, 90%, and 50% of the target volume respectively were studied. Results: The median prescribed dose was 68 Gy (60 Gy-72 Gy). The median percentage D95 for 3mm, 5mm and 7mm VOU was 82.5% ± 4.95, 77.25% ± 5.53, and 69% ± 6.93, respectively. The median percentage D90 for these VOU's was 87.7% ± 3.53, 83.2% ± 4.61, and 79% ± 4.5, respectively. The median dose to each of these VOU”s was 96% ± 1.6, 94.5% ± 1.95, and 92.5% ± 1.85 respectively. Conclusions: This study documents that the volumes of uncertainty surrounding the PTV, which could contain subclinical disease, in fact receive a significant amount of RT dose. Hence, despite a large amount of evidence for inter-observer variability in target delineation for head and neck cancer,the majority of locoregional recurrences are within the high dose region and not marginal failures. No significant financial relationships to disclose.

Impact of interobserver variability in contouring of AVMs

Delineation of the nidus of brain AVMs on digital subtraction angiography (DSA) forms the basis of target definition for radiosurgery (RS). However, interclinician discrepancy in the assessment of nidus position and volume exists. In this study, the magnitude of the interobserver variability as well as its dosimetric and clinical impact are assessed. DSAs from 31 patients who underwent RS for a brain AVM were selected for nidus contouring by six experienced clinicians (2 neurosurgeons, 2 radiation oncologists, 2 neuroradiologists) using BrainScan v5.2 software (BrainLAB AG, Heimstetten, Germany). All patients had a minimum follow-up of 3 years or earlier complete obliteration. The interobserver variability was assessed by 1. the ratio between the volume of agreement (VOA6) of all observers and the corresponding encompassing volume (ECV); 2. the same ratio for all possible pairs of observers (VOA2/ECV2); 3. translation of the center of mass of all individual volumes. Dosimetric consequences were calculated for both the use of standard collimators and for dynamic conformal arcs. The original treatment plan was evaluated for dosimetric coverage of the VOA6. In one patient no VOA6 could be determined due to a large contouring variation. The mean ratio between VOA6 and ECV6 was 0.19 ± 0.14. For all possible pairs of clinicians, the VOA6/ECV6 was 0.45 ± 0.14. The mean translation between the center of mass of the individual volumes was 2.6 mm. The mean coverage of the individual volumes by the 80% ( = prescription isodose)and 75% isodoses was 88.1% and 90.2%, respectively, when standard round collimators were used. Using dynamic arc technique, this was significantly (p = 0.002) lower (78.9% and 82,6%, respectively). Complete obliteration was seen in 24/31 (77%) cases at a mean of 33 months. There was no significant relationship with nidus volume (p = 0.09) and marginal SRS dose (p = 0.09). The VOA6 was incompletely covered by the original treatment plan in 3/31 patients (10%); In 1/24 with complete obliteration (4%) and 2/7 with incomplete obliteration (29%). Failure to obliterate did not correlate with VOA6/ECV6 (p = 0.69), VOA2/ECV2 (p = 0.56), but was significantly correlated with the mean coverage of individual volumes by the 80% isodose (p = 0.004). There is a significant interobserver variability in the contouring of the nidus on DSA and a better definition is required. This variability may result in underdosage. The clinical impact of this will increase with the use of more conformal stereotactic techniques

Impact of interobserver variability in contouring of AVMs

Delineation of the nidus of brain AVMs on digital subtraction angiography (DSA) forms the basis of target definition for radiosurgery (RS). However, interclinician discrepancy in the assessment of nidus position and volume exists. In this study, the magnitude of the interobserver variability as well as its dosimetric and clinical impact are assessed.

The impact of contouring uncertainty on rectal 3D dose–volume data: Results of a dummy run in a multicenter trial (AIROPROS01–02)

Purpose To estimate the impact of the uncertainty in contouring the rectum on rectal dose–volume parameters and normal tissue complication probability (NTCP) in a prospective (AIROPROS01-02) investigation about rectal toxicity. Methods and materials The participants in a prospective trial (18 observers) were asked to draw the external contour of the rectum of 4 sample patients (3 patients undergoing radical conformal radiotherapy, 1 patient undergoing post-prostatectomy) on CT images (0.5 cm spacing) using a 3D treatment planning system. A previously accepted definition of cranial and caudal borders of the rectum was applied. For each patient, four- and six-field 3D-conformal techniques (70–76 Gy, ICRU dose) were planned and DVH/dose statistics of the rectum were calculated. The impact of interobserver variability on rectal volume, cranial and caudal borders, mean, maximum, and median rectal dose, percentage of rectum receiving more than 40, 45, 50, 55, 60, 65, 70, and 75 Gy (V 40–V 75), and NTCP were investigated. Results Concerning DVHs, 9/18 observers tended to have some systematic deviation. However, deviations from the mean values greater than 5% were found only in 1/9 because of a systematic discrepancy in the caudal limit assessment (mean deviation from the most frequently chosen slice: 8 mm). No other observers showed a mean deviation in the cranial or the caudal limit definition greater than 5.8 mm. For another observer, it was possible to clearly assess the cause of a relatively large systematic deviation for DVH parameters. In both cases, the observers were contacted to avoid these systematic deviations. When considering the remaining 16/18 observers, the average values of SD for V 40–V 75 ranged between 1% and 4% and were found to be lower (<3%) for the 3 nonoperated patients. The average values of the SD were around 1.5–2 Gy and less than 1.5% for mean/median dose and NTCPs, respectively. Conclusions Concerning the uncertainty in rectum definition, the collection of rectal dose–volume data in multicenter investigations seems to be feasible after a clear and previously accepted definition of rectum is assessed. However, even with a general agreement on rectum definition, contouring appears to be a quite significant source of uncertainty. A dummy run procedure is useful in identifying possible discrepancies among single observers and in assessing reliable confidence levels on dose–volume constraints because of contouring uncertainty, making the dummy run mandatory in multicenter trials evaluating 3D dose–volume data.

Rectum contouring variability in patients treated for prostate cancer: impact on rectum dose–volume histograms and normal tissue complication probability

Background: Recent investigations showed some correlation between three-dimensional (3D) treatment planning dose–volume data (dose–volume histograms: DVH, dose statistics) and rectal toxicity for patients treated for prostate cancer. However, no data are available about the possible impact of inter-institute variability in contouring the rectum, so that the possibility of reliably using information from single-centre studies remains doubtful. Purpose: Within a retrospective three-institutes study on correlation between dose–volume treatment planning data and rectum bleeding in patients treated for prostate cancer, an investigation about the impact of inter- and intra-observer variability in contouring the rectum was performed. Materials and methods: Ten patients were considered for a dummy run exercise and three observers (one per Institute) contoured the rectum (including filling). An anatomically based definition of rectum extension was previously accepted by the three observers. Six of the ten patients were randomly chosen in the subgroup of patients (large spacing, LS) with a distance between computed tomography (CT) slices (outside the prostate region) equal to 10 mm; for the remaining four patients the distance between CT slices was 5 mm over the whole rectum volume (small spacing, SS). The original 3D treatment planning was recovered on the Cadplan treatment planning system for each patient and rectum dose statistics (mean, median and maximum rectum dose), volume, DVH and NTCP values were calculated for each observer. For DVH analysis, the values of V 50, V 55, V 60, V 65 and V 70 (defined as the % of rectum volume receiving at least 50, 55, 60, 65, 70 Gy) were considered. Normal tissue complication probabilities (NTCPs) were calculated for the original ICRU dose and for a 75.6 Gy ICRU dose (NTCP and NTCP 75.6, respectively). Intra-observer variability was investigated by asking the observers to redraw the same rectum contours 6 months later and comparing the two contouring sessions. Results: In general, a good agreement was found for most patients and, in particular, for all SS patients. The impact of inter-observer variability was quite significant on dose statistics and DVH in two of six LS patients. Looking at the patient population, some systematic deviations, even if quite small, were demonstrated between institute B and institute C (volume, P=0.02) and between institute A and institute B (mean/median dose, V 50–V 65, NTCP 75.6; P<0.05). Four of six LS patients (0/4 in the SS group) presented a maximum difference among observers at the cranial and/or caudal limit of the rectum equal to 1 cm. For these patients, inter-observer variability was significantly higher than for the others ( P<0.03). When inter-observer variability was expressed in terms of standard deviations (SD), values around 2–3 Gy and 0.5 Gy for LS and SS patients, respectively, were found for mean/median dose; values around 3–4% and 0.5–2% for LS and SS patients, respectively, were found for V 50– V 70. The average SD for NTCP and NTCP 75.6 were 0.4 and 0.6%, respectively (0.5 and 0.9% for LS patients; 0.2 and 0.3% for SS patients). Intra-observer variability was found to be lower than inter-observer variability even if the impact on dose statistics and DVH was visible. Conclusions: Once a robust definition of rectum is assessed, inter- and intra-institute variability in contouring the rectum appear relatively modest. However, the results suggest that the number of LS patients in DVH correlation studies should be as low as possible; the low number of these patients in the multi-centric trial involving our institutions should not have significant impact on the results of the study.

Evaluation of permanent prostate implants: a pilot study comparing dosimetric impact of inter-observer variability using CT and MRI

Evaluation of permanent prostate implants: a pilot study comparing dosimetric impact of inter-observer variability using CT and MRI

Technetium-99m-pyrophosphate scintigraphy in patients with suspected acute myocardial infarction: Impact of interobserver variability

Technetium-99m-pyrophosphate (TcPYP) scintigraphy may have great value in patients with suspected acute myocardial infarction (AMI), but interobserver variability undoubtedly has adverse impact on predictive value. TcPYP scintigrams for 133 (80%) of 166 consecutive patients admitted for suspected AMI were interpreted independently by three experienced readers. Although there was complete agreement for 87 interpretations (65%), major discrepancies (i.e., at least one positive and one negative reading on the same scan) occurred for 28 scans (21%). To assess predictive accuracy, patients were categorized as follows: 36 had definite AMI manifest by new ECG Q waves and/or CK-MB evidence of AMI (group I), 56 were classified as possible AMI (group II), and 41 had AMI excluded (group III). Using only the definitive diagnostic categories (groups I and III), accuracy for each reader approximated 0.68, with no single reader being correct more often than any other.