Reduction In Interobserver Variability Research Articles

Revisiting the utility of identifying nuclear grooves as unique nuclear changes by an object detector model.

Among other structures, nuclear grooves are vastly found in papillary thyroid carcinoma (PTC). Considering that the application of artificial intelligence in thyroid cytology has potential for diagnostic routine, our goal was to develop a new supervised convolutional neural network capable of identifying nuclear grooves in Diff-Quik stained whole-slide images (WSI) obtained from thyroid fineneedle aspiration. We selected 22 Diff-Quik stained cytological slides with cytological diagnosis of PTC and concordant histological diagnosis. Each of the slides was scanned, forming a WSI. Images that contained the region of interest were obtained, followed by pre-formatting, annotation of the nuclear grooves and data augmentation techniques. The final dataset was divided into training and validation groups in a 7:3 ratio. This is the first artificial intelligence model based on object detection applied to nuclear structures in thyroid cytopathology. A total of 7,255 images were obtained from 22 WSI, totaling 7,242 annotated nuclear grooves. The best model was obtained after it was submitted 15 times with the train dataset (14th epoch), with 67% true positives, 49.8% for sensitivity and 43.1% for predictive positive value. The model was able to develop a structure predictor rule, indicating that the application of an artificial intelligence model based on object detection in the identification of nuclear grooves is feasible. Associated with a reduction in interobserver variability and in time per slide, this demonstrates that nuclear evaluation constitutes one of the possibilities for refining the diagnosis through computational models.

Clinical Evaluation of an Artificial Intelligence-Based Decision Support System for the Diagnosis and American College of Radiology Thyroid Imaging Reporting and Data System Classification of Thyroid Nodules.

Background: This study aimed to evaluate the clinical impact of an artificial intelligence (AI)-based decision support system (DSS), Koios DS, on the analysis of ultrasound imaging and suspicious characteristics for thyroid nodule risk stratification. Methods: A retrospective ultrasound study was conducted on all thyroid nodules with histological findings from June 2021 to December 2022 in a thyroid nodule clinic. The diagnostic performance of ultrasound imaging was evaluated by six readers on the American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) before and after the use of the AI-based DSS and by AI itself. Results: A total of 172 patients (83.1% women) with a mean age of 52.3 ± 15.3 years were evaluated. The mean maximum nodular diameter was 2.9 ± 1.2 cm, with 11.0% being differentiated thyroid carcinomas. Among the nodules initially classified as ACR TI-RADS 3 and 4, AI reclassified 81.4% and 24.5% into lower risk categories, respectively. Receiver operating characteristic (ROC) curve analysis was performed to evaluate the diagnostic performance of the readers and the AI-based DSS versus histological diagnosis. There was an increase in the area under the ROC curve (AUROC) after the use of AI (0.776 vs. 0.817, p < 0.001). The AI-based DSS improved the mean sensitivity (Sens) (82.3% vs. 86.5%) and specificity (Spe) (38.3% vs. 54.8%), produced a high negative predictive value (94.5% vs. 96.4%), and increased the positive predictive value (PPV) (14.0% vs. 16.1%) and diagnostic precision (43.0% vs. 49.3%). Based on the ACR TI-RADS score, there was significant improvement in interobserver agreement after the use of AI (r = 0.741 for ultrasound imaging alone vs. 0.981 for ultrasound imaging and the AI-based DSS, p < 0.001). Conclusions: The use of an AI-based DSS was associated with overall improvement in the diagnostic efficacy of ultrasound imaging, based on the AUROC, as well as an increase in Sens, Spe, negative and PPVs, and diagnostic accuracy. There was also a reduction in interobserver variability and an increase in the degree of concordance with the use of AI. AI reclassified more than half of the nodules with intermediate ACR TI-RADS scores into lower risk categories.

Clinical validation of an AI-based automatic quantification tool for lung lobes in SPECT/CT

BackgroundLung lobar ventilation and perfusion (V/Q) quantification is generally obtained by generating planar scintigraphy images and then imposing three equally sized regions of interest on the data of each lung. This method is fast but not as accurate as SPECT/CT imaging, which provides three-dimensional data and therefore allows more precise lobar quantification. However, the manual delineation of each lobe is time-consuming, which makes SPECT/CT incompatible with the clinical workflow for V/Q estimation. An alternative may be to use artificial intelligence-based auto-segmentation tools such as AutoLung3D (Siemens Healthineers, Knoxville, USA), which automatically delineate the lung lobes on the CT data acquired with the SPECT data. The present study assessed the clinical validity of this approach relative to planar scintigraphy and manual quantification in SPECT/CT.MethodsThe Autolung3D software was tested on the retrospective SPECT/CT data of 43 patients who underwent V/Q scintigraphy with 99mTc-macroaggregated albumin and 99mTc-labeled aerosol. It was compared to planar scintigraphy and SPECT/CT using the manual quantification method in terms of relative lobar V/Q quantification values and interobserver variability.ResultsThe three methods provided similar V/Q estimates for the left lung lobes and total lungs. However, compared to the manual SPECT/CT method, planar scintigraphy yielded significantly higher estimates for the middle right lobe and significantly lower estimates for the superior and inferior right lobes. The estimates of the manual and automated SPECT/CT methods were similar. However, the post-processing time in the automated method was approximately 5 min compared to 2 h for the manual method. Moreover, the automated method associated with a drastic reduction in interobserver variability: Its maximal relative standard deviation was only 5%, compared to 23% for planar scintigraphy and 19% for the manual SPECT/CT method.ConclusionsThis study validated the AutoLung3D software for general clinical use since it rapidly provides accurate lobar quantification in V/Q scans with markedly less interobserver variability than planar scintigraphy or the manual SPECT/CT method.

Clinical validation of deep learning algorithms for radiotherapy targeting of non-small-cell lung cancer: an observational study

Artificial intelligence (AI) and deep learning have shown great potential in streamlining clinical tasks. However, most studies remain confined to in silico validation in small internal cohorts, without external validation or data on real-world clinical utility. We developed a strategy for the clinical validation of deep learning models for segmenting primary non-small-cell lung cancer (NSCLC) tumours and involved lymph nodes in CT images, which is a time-intensive step in radiation treatment planning, with large variability among experts. In this observational study, CT images and segmentations were collected from eight internal and external sources from the USA, the Netherlands, Canada, and China, with patients from the Maastro and Harvard-RT1 datasets used for model discovery (segmented by a single expert). Validation consisted of interobserver and intraobserver benchmarking, primary validation, functional validation, and end-user testing on the following datasets: multi-delineation, Harvard-RT1, Harvard-RT2, RTOG-0617, NSCLC-radiogenomics, Lung-PET-CT-Dx, RIDER, and thorax phantom. Primary validation consisted of stepwise testing on increasingly external datasets using measures of overlap including volumetric dice (VD) and surface dice (SD). Functional validation explored dosimetric effect, model failure modes, test-retest stability, and accuracy. End-user testing with eight experts assessed automated segmentations in a simulated clinical setting. We included 2208 patients imaged between 2001 and 2015, with 787 patients used for model discovery and 1421 for model validation, including 28 patients for end-user testing. Models showed an improvement over the interobserver benchmark (multi-delineation dataset; VD 0·91 [IQR 0·83-0·92], p=0·0062; SD 0·86 [0·71-0·91], p=0·0005), and were within the intraobserver benchmark. For primary validation, AI performance on internal Harvard-RT1 data (segmented by the same expert who segmented the discovery data) was VD 0·83 (IQR 0·76-0·88) and SD 0·79 (0·68-0·88), within the interobserver benchmark. Performance on internal Harvard-RT2 data segmented by other experts was VD 0·70 (0·56-0·80) and SD 0·50 (0·34-0·71). Performance on RTOG-0617 clinical trial data was VD 0·71 (0·60-0·81) and SD 0·47 (0·35-0·59), with similar results on diagnostic radiology datasets NSCLC-radiogenomics and Lung-PET-CT-Dx. Despite these geometric overlap results, models yielded target volumes with equivalent radiation dose coverage to those of experts. We also found non-significant differences between de novo expert and AI-assisted segmentations. AI assistance led to a 65% reduction in segmentation time (5·4 min; p<0·0001) and a 32% reduction in interobserver variability (SD; p=0·013). We present a clinical validation strategy for AI models. We found that in silico geometric segmentation metrics might not correlate with clinical utility of the models. Experts' segmentation style and preference might affect model performance. US National Institutes of Health and EU European Research Council.

The impact of an educational tool in cervix image registration across three imaging modalities.

Accurate image registration is vital in cervical cancer where changes in both planning target volume (PTV) and organs at risk (OARs) can make decisions regarding image registration complicated. This work aims to determine the impact of a dedicated educational tool compared with experience gained in MR-guided radiotherapy (MRgRT). 10 therapeutic radiographers acted as observers and were split into two groups based on previous experience with MRgRT and Monaco treatment planning system. Three CBCT-CT, three MR-CT and two MR-MR registrations were completed per patient by each observer. Observers recorded translations, time to complete image registration and confidence. Data were collected in two phases; prior to and following the introduction of a cervix registration guide. No statistically significant differences were noted between imaging modalities. Each group was assessed independently pre- and post-education, no statistically significant differences were noted in either CBCT-CT or MR-CT imaging. Group 1 MR-MR imaging showed a statistically significant reduction in interobserver variability (p=0.04), in Group 2, the result was not statistically significant (p=0.06). Statistically significant increases in confidence were seen in all three modalities (p≤0.05). At The Christie NHS Foundation Trust, radiographers consistently registered images across three different imaging modalities regardless of their previous experience. The implementation of an image registration guide had limited impact on inter- and intraobserver variability. Radiographers' confidence showed statistically significant improvements following the use of the registration manual. This work helps evaluate training methods for novel roles that are developing in MRgRT.

Contrast-Sparing Imaging Utilizing Spectral Detector CT for Transcatheter Aortic Valve Replacement Procedure Planning

ABSTRACT Background : Cardiac CTA is an indispensable imaging tool for TAVR procedure planning. Post-processing of the dual-energy CT utilizing dual-source and kV-switching approach enables an increase in the density of iodine and allows the reduction of iodine dose. We hypothesized that the use of a dual-layer Spectral Detector CT (SDCT) can enhance the signal of intravascular iodine contrast material, reduce iodine contrast material volume, and facilitate pre-TAVR planning. Methods : We tested this in a preclinical porcine animal model, with results suggesting that spectral imaging may be superior to conventional imaging even at 21-35% of the full contrast medium dose with regard to reader confidence, higher SNR and CNR at the level of the aortic annulus and root. We subsequently followed this up by a prospective human validation study of 24 patients undergoing TAVR. Results : We demonstrate that SNR and CNR with SDCT were significantly higher (highest in lower energy virtual mono-energetic images (VMI) (monoE 40 keV) compared to conventional images, with the spectral images preferred for procedure planning by an experienced TAVR operator and imaging specialist. This was associated with a reduction in inter-observer variability in TAVR sizing measurements in low dose contrast studies (33% of the full contrast dose) resulting in a higher rate of agreement on the choice of valve prosthesis size. Conclusion : Low contrast dose spectral images achieved similar SNR and CNR compared to full contrast dose conventional images. Taken together, our results suggest that the use of SDCT imaging may facilitate the routine use of low contrast dose as part of pre-TAVR imaging.

A teaching intervention in a contouring dummy run improved target volume delineation in locally advanced non-small cell lung cancer: Reducing the interobserver variability in multicentre clinical studies.

Interobserver variability in the definition of target volumes (TVs) is a well-known confounding factor in (multicentre) clinical studies employing radiotherapy. Therefore, detailed contouring guidelines are provided in the prospective randomised multicentre PET-Plan (NCT00697333) clinical trial protocol. This trial compares strictly FDG-PET-based TV delineation with conventional TV delineation in patients with locally advanced non-small cell lung cancer (NSCLC). Despite detailed contouring guidelines, their interpretation by different radiation oncologists can vary considerably, leading to undesirable discrepancies in TV delineation. Considering this, as part of the PET-Plan study quality assurance (QA), a contouring dummy run (DR) consisting of two phases was performed to analyse the interobserver variability before and after teaching. In the first phase of the DR (DR1), radiation oncologists from 14 study centres were asked to delineate TVs as defined by the study protocol (gross TV, GTV; and two clinical TVs, CTV-A and CTV-B) in a test patient. A teaching session was held at a study group meeting, including a discussion of the results focussing on discordances in comparison to the per-protocol solution. Subsequently, the second phase of the DR (DR2) was performed in order to evaluate the impact of teaching. Teaching after DR1 resulted in a reduction of absolute TVs in DR2, as well as in better concordance of TVs. The Overall Kappa(κ) indices increased from 0.63 to 0.71 (GTV), 0.60 to 0.65 (CTV-A) and from 0.59 to 0.63 (CTV-B), demonstrating improvements in overall interobserver agreement. Contouring DRs and study group meetings as part of QA in multicentre clinical trials help to identify misinterpretations of per-protocol TV delineation. Teaching the correct interpretation of protocol contouring guidelines leads to a reduction in interobserver variability and to more consistent contouring, which should consequently improve the validity of the overall study results.

A Prospective Study to Determine Inter-observer Variability of Gross Tumor Volume with [18F] Fludeoxyglucose-PET/CT Compared to CT Alone in Stage III Non-Small Cell Lung Cancer Using Three-Dimensional Analysis

Purpose: This study determined the interobserver variability of Gross Tumor Volume (GTV) with [18F] Fludeoxyglucose (FDG)-PET/CT compared to CT alone in Stage III Non-Small Cell Lung Cancer (NSCLC) using 3D analysis. Materials and Methods: Twenty-nine patients underwent simultaneous co-registered CT and FDG-PET/CT for radiotherapy planning. GTV for lung tumor and mediastinal lymphadenopathy contoured by three different radiation oncologists was compared for changes in volume and position. Interobserver variability was determined with vector displacement and Dice Similarity Coefficient (DSC). Concordance for the number of lymph nodes was determined. Results: Mean GTV for lung tumor with FDG-PET/CT and CT alone was 62.0 cm3 and 74.64 cm3, (p=0.0005) with 17% reduction in GTV by FDG-PET/CT. Mean GTV for mediastinal lymphadenopathy was 15.72 cm3 and 19.02 cm3 (p=0.084) with 17% reduction in GTV. Mean vector displacement of lung tumor was 2.0 mm with FDG-PET/CT versus 7.1 mm with CT alone (p = 0.0016) with 3.6 fold reduction in interobserver variability. Mean vector displacement of mediastinal lymphadenopathy was 1.53 mm with FDG-PET versus 10.2 mm for CT alone (p= 0.0005) with 6.7 fold reduction in interobserver variability. Median DSC for the primary GTV was 0.87 for FDG-PET/CT and 0.74 for CT alone. Median DSC for nodal GTV was 0.79 and 0.59 respectively. All physicians agreed on the number of lymph nodes on CT alone in 15/29 vs. 27/29 patients on PET/CT. Only two of the three physicians agreed on the number of lymph nodes contoured for CT alone in 12/29 versus only 2/29 patients for FDG-PET/CT (p=0.0018). Conclusion: FDG-PET/CT compared to CT alone is more precise, reduces mean lung tumor and mediastinal nodal GTV and interobserver variability. There was greater agreement for the number of lymph nodes contoured on FDG-PET/CT compared to CT alone.

Reduction of interobserver variability (IOV) by FDG-PET–based tumor volume delineation (TVD) in radiotherapy planning for pancreatic cancer.

229 Background: Wide IOV has been reported for CT-based TVD during radiotherapy planning for pancreatic cancer (Yamazaki, Anticancer Research 27: 2965-2972, 2007). PET-CT has been successfully incorporated into treatment planning in other tumour sites and has been shown to reduce IOV, but no study has looked at its role in pancreatic radiotherapy. Methods: Six radiation oncologists from 6 oncology centers in the UK were asked to outline GTVs in two cases with pancreatic head tumors. DICOM data set of planning CT, diagnostic CT and FDG-PET-CT scans were made available. GTV was first outlined using the information on the diagnostic CT scan alone, without access to PET scans (GTV_CT). The GTVs were then re-outlined using additional information from the PET scan (GTV_PET). A PTV was grown from the GTV, using a margin of 2cm sup-inf and 1.5cm radially from the GTV, as per standard UK practice. Results: A total of 48 countours (24 GTV contours and 24 PTV contours) were analysed, and the results are presented below (Table). Conclusions: The incorporation of PET scan resulted in a prominent reduction in standard deviation around tumor volumes, smaller largest:smallest GTV ratios and a larger percentage of overlap between investigator volumes, suggesting a reduction in Inter-Observer Variability. PET based RT planning should be considered in multicenter radiotherapy trials in pancreatic cancer to reduce IOV. [Table: see text]

Reproducibility of Dynamic Contrast-enhanced MR Imaging. Part II. Comparison of Intra- and Interobserver Variability with Manual Region of Interest Placement versus Semiautomatic Lesion Segmentation and Histogram Analysis

To compare the inter- and intraobserver variability with manual region of interest (ROI) placement versus that with software-assisted semiautomatic lesion segmentation and histogram analysis with respect to quantitative dynamic contrast material-enhanced (DCE) MR imaging determinations of the volume transfer constant (K(trans)). The study was approved by the institutional review board and compliant with HIPAA. The requirement to obtain informed consent was waived. Fifteen DCE MR imaging studies of the female pelvis defined the study group. Uterine fibroids were used as a perfusion model. Three varying types of lesion measurements were performed by five readers on each study by using DCE MR imaging perfusion analysis software with manual ROI placement and a semiautomatic lesion segmentation and histogram analysis solution. Intra- and interreader variability of measurements of K(trans) with the different measurement types was calculated. The overall interobserver variability of K(trans) with manual ROI placement (mean, 28.5% ± 9.3) was reduced by 42.5% when the semiautomatic, software-assisted lesion measurement method was used (16.4% ± 6.2). Whole-lesion measurement showed the lowest interobserver variability with both measurement methods (20.1% ± 4.3 with the manual method vs 10.8% ± 2.6 with the semiautomatic method). The overall intrareader variability with the manual ROI method (7.6% ± 10.6) was not significantly different from that with the semiautomatic method (7.3% ± 10.8), but the intraclass correlation coefficient for intrareader reproducibility improved from 0.86 overall with the manual method to 0.99 with the semiautomatic method. A semiautomatic lesion segmentation and histogram analysis approach can provide a significant reduction in interobserver variability for DCE MR imaging measurements of K(trans) when compared with manual ROI methods, whereas intraobserver reproducibility is improved to some extent.

Can a Teaching Intervention Reduce Interobserver Variability in LVEF Assessment: A Quality Control Exercise in the Echocardiography Lab

This study sought to determine whether a formalized teaching intervention could reduce the interobserver variability (IOV) in visual estimation of left ventricular ejection fraction (LVEF) within a group of sonographers and physicians with a spectrum of experience. Precise and reliable echocardiographic assessment of LVEF is necessary for clinical decision-making and minimizing duplicative testing. Skill in the visual estimation of LVEF varies depending on experience and is critical for corroborating EF quantification. IOV may also lead to inconsistency if multiple readers are assessing the EF on serial exams. Fourteen cases of 2-dimensional echocardiograms were shown to 25 participants who estimated the EF based on a complete assessment of LV wall motion including parasternal, short-axis, apical, and subcostal views. The cases represented a spectrum of EF range, image quality, and clinical context. Following the initial interpretations, participants underwent a teaching intervention involving tutorial review of reference cases and group discussion of each case with determination of the EF guided by quantitative measure (biplane Simpson method). Three months after the teaching intervention, 14 new cases were shown to the 25 participants following the same methodology. IOV was quantified before and after the teaching intervention with the use of a 3-factor, nested analysis of variance. The factors were: observer, patient, and pre- and post-intervention (time). The analysis of variance showed that the intervention reduced the IOV for the 25 readers between the pre- and post-intervention assessments (F = 2.8, p = 0.007). The IOV decreased from ± 14% EF prior to intervention to ± 8.4% EF following intervention (a 40% reduction in IOV). In a large echocardiography laboratory with a wide range of training levels and experience, a simple, formalized teaching intervention can successfully diminish IOV of LVEF assessment. This intervention provides not only discrete quality measures, but also serves as a practical tool to document and improve quality of reporting, potentially reducing clinical inefficiencies and repeat testing.

Contrast Echocardiography in Australian Clinical Practice

The second-generation contrast agent Definity (a perflutren microsphere) became available in Australia in mid-2007. We describe the introduction of contrast echocardiography into a high-volume quaternary teaching hospital, performing over 16,000 echocardiograms per year. Workflow protocols were developed for patient selection, contrast administration, and image acquisition and analysis. Data were prospectively collected for all contrast cases. Endocardial definition scores were derived by three independent observers before and after contrast administration, and statistically compared. 161 patients received contrast in the first 12 months of the contrast program. There was statistically significant improvement in endocardial definition scores after contrast administration (p=0.0001), and reduction in inter-observer variability of wall motion assessment. A number of clinically significant findings (pseudoaneurysm, non-compaction, thrombus) were detected on contrast echo that were not apparent on standard 2D imaging. Adverse events were rare (0.6%) with no life-threatening events. The introduction of a second-generation contrast agent into clinical workflow in a hospital echocardiography department resulted in a statistically significant improvement in endocardial definition, and safely provided diagnostic imaging in cases which were otherwise non-diagnostic. Inter-observer variability was reduced, and diagnostic yield increased. These results reflect previously published data, and indicate that contrast echocardiography is feasible in Australian clinical practice.

Automated left ventricular delineation in X‐ray angiograms: A validation study

Recently an automated analysis approach for left ventricular (LV) X-ray angiographic studies was proposed. This particular study aims to assess the clinical potential of this approach. Over the past 30 years much research has been carried out to develop a technique with automated contour detection of the LV outline in the end-diastolic (ED) and end-systolic (ES) phases. Very few have made it into clinical practice. Our latest approach is based on innovative model-based image processing techniques. Two expert cardiologists analyzed 30 patient studies both by contouring the LV manually and by using the proposed automated methodology. In the latter procedure the experts were allowed to edit the automatically generated contours manually. The manual, automatic, and edited automatic contours were compared, focusing on accuracy, workflow efficiency, and inter- and intra-observer variabilities. No significant differences between the automatically derived and manual LV volumes were observed. The average patient study analysis time was reduced by 26%, from 4.2 to 3.1 min. When editing was required, 19% of the ED and 25% of the ES contour length needed manual correction. Furthermore, a reduction in inter-observer variability of 12.4% was observed. Using the proposed automated methodology for X-ray LV angiographic study analysis, a considerable reduction in required analysis time and manual effort is achieved. Because the acquired results are of clinically acceptable quality and the inter- and intra-observer variabilities are reduced, this automated approach has the potential to optimize the analysis workflow for LV X-ray angiography in clinical practice.