Interobserver Variation Research Articles

A deep learning based method for left ventricular strain measurements: repeatability and accuracy compared to experienced echocardiographers

Abstract Background Speckle tracking echocardiography (STE) provides quantification of left ventricular (LV) deformation and is useful in the assessment of LV function. STE is increasingly being used clinically, and every effort to simplify and standardize STE is important. Manual outlining of regions of interest (ROIs) is labor intensive and may influence assessment of strain values. Purpose We hypothesized that a deep learning (DL) model, trained on clinical echocardiographic exams, can be combined with a readily available echocardiographic analysis software, to automate strain calculation with comparable fidelity to trained cardiologists. Methods Data consisted of still frame echocardiographic images with cardiologist-defined ROIs from 672 clinical echocardiographic exams from a university hospital outpatient clinic. Exams included patients with ischemic heart disease, heart failure, valvular disease, and conduction abnormalities, and some healthy subjects. An EfficientNetB1-based architecture was employed, and different techniques and properties including data set size, data quality, augmentations, and transfer learning were evaluated. DL predicted ROIs were reintroduced into commercially available echocardiographic analysis software to automatically calculate strain values. Results DL-automated strain calculations had an average absolute difference of 0.75 (95% CI 0.58–0.92) for global longitudinal strain (GLS), and 1.16 (95% CI 1.03–1.29) for single-projection longitudinal strain (LS), compared to operators. A Bland–Altman plot revealed no obvious bias, though there were fewer outliers in the lower average LS ranges. Techniques and data properties yielded no significant increase/decrease in performance. Conclusion The study demonstrates that DL-assisted, automated strain measurements are feasible, and provide results within interobserver variation. Employing DL in echocardiographic analyses could further facilitate adoption of STE parameters in clinical practice and research, and improve reproducibility.

Automated acute skin toxicity scoring in a mouse model through deep learning.

This study presents a novel approach to skin toxicity assessment in preclinical radiotherapy trials through an advanced imaging setup and deep learning. Skin reactions, commonly associated with undesirable side effects in radiotherapy, were meticulously evaluated in 160 mice across four studies. A comprehensive dataset containing 7542 images was derived from proton/electron trials with matched manual scoring of the acute toxicity on the right hind leg, which was the target area irradiated in the trials. This dataset was the foundation for the subsequent model training. The two-step deep learning framework incorporated an object detection model for hind leg detection and a classification model for toxicity classification. An observer study involving five experts and the deep learning model, was conducted to analyze the retrospective capabilities and inter-observer variations. The results revealed that the hind leg object detection model exhibited a robust performance, achieving an accuracy of almost 99%. Subsequently, the classification model demonstrated an overall accuracy of about 85%, revealing nuanced challenges in specific toxicity grades. The observer study highlighted high inter-observer agreement and showcased the model's superiority in accuracy and misclassification distance. In conclusion, this study signifies an advancement in objective and reproducible skin toxicity assessment. The imaging and deep learning system not only allows for retrospective toxicity scoring, but also presents a potential for minimizing inter-observer variation and evaluation times, addressing critical gaps in manual scoring methodologies. Future recommendations include refining the system through an expanded training dataset, paving the way for its deployment in preclinical research and radiotherapy trials.

Measuring up: Ensuring Intra- and Interobserver Reliability in Stretched Penile Length with the SPLINT Technique

ABSTRACT Background: A discrepancy between the true and measured value of stretched penile length (SPL) may be a result of errors that can either be systematic or random. Hence, it becomes important to focus on the quality of measurements to prevent any iatrogenic harm to the patients. Objective: The objective of this study was to assess the magnitude of intra- and interobserver variations in the measurement of SPL with the SPLINT technique. Materials and Methods: SPL was measured prospectively in a cohort of 449 boys aged 0–14 years including 68 infants (substratified into Group I: >4 years, Group II: 4–8 years, and Group III: >8 years) with the SPLINT technique by expert (E: E1 and E2) and trainee (T: T1 and T2) surgeons after completing a three-tiered training module. Intra- and interobserver variability was assessed through descriptive statistics, intraclass correlation (ICC), relative technical error of measurement (rTEM), and reliability or R (%). Results: Intraobserver variability: the mean difference between the two readings (E1 and E2) is 0.08 cm (95% confidence interval [CI]: 0.073–0.087), ICC was 0.998 (95% CI: 0.997–0.998), and intraobserver variability ≤0.1 cm in 85% of the participants (n = 370 of 433). The rTEM and reliability (%) were 1.82% and 98.1% (Group I), 1.65% and 98.9% (Group II), and 1.09% and 99.7% (Group III), respectively. The intraobserver variability was observed to be inversely proportional to the age of the participants (correlation coefficient = −0.56). Interobserver variability was calculated separately for expert versus trainee and trainee versus trainee (T-vs-T) measurements. For expert versus expert, ICC, rTEM, and reliability (%) were 0.984, 2.4%, and 96.8% (Group 1), 0.992, 2.07%, and 98.3% (Group 2), and 0.997, 1.38%, and 99.05% (Group 3), respectively. A similar pattern of variability was observed for T-vs-T measurements. The reliability (%) of the SPL by experts is consistently more than that of trainees across all age groups; however, the difference ameliorates with the age of participant. Conclusions: The study has validated the SPLINT technique by demonstrating a high level of intra- and interobserver reliability. The adequacy of the training modules for SPL measurements described in this study has also been established. Evidence that the SPL can be used as an objective marker of penile dimensions is herewith furnished.

Apparatus and method for measuring individual color-matching functions

This article presents a compact visual colorimeter for the purpose of measuring color-matching functions (CMFs) of individual observers through psychophysical experiments. Constructed from 3D-printed parts, optical elements, and LED-based light engines, the colorimeter facilitates the juxtaposition of two fields to create a bipartite field. The system underwent characterization to evaluate factors that may impact color-matching experiments, such as LED-light stability, spatial homogeneity of the bipartite field, and potential stray-light leakage. The study aimed to assess the accuracy and performance of the system in measuring individual observer CMFs. Results indicate that the system is stable enough to measure both intra- and inter-observer variations in CMFs.

Three-dimensional source position verification in image-guided high-dose-rate brachytherapy using an XCT-based gel dosimeter.

Comprehensive quality assurance (QA) for a seamless workflow of high-dose-rate brachytherapy, from imaging to planning and irradiation, is uncommon, and QA of the source dwell position is performed in one- or two-dimensions. Gel dosimetry using magnetic resonance imaging (MRI) is effective in verifying the three-dimensional distribution of doses for image-guided brachytherapy (IGBT). However, MRI scanners are not readily accessible, and MRI scanning is time-consuming. Nevertheless, X-ray computed tomography (XCT) is available for IGBT planning. In this study, we designed and developed an efficient method for QA for a seamless workflow of IGBT with a new commercially available XCT-based polymer gel dosimeter. To enable direct insertion of brachytherapy applicators, the gelatinizing agent of the dosimeter was modified. A cylindrical polyvinyl chloride jar was filled with the modified gel dosimeter, which was subsequently used to determine the reproducibility of source dwell positions, detectability of source positional errors from intentionally introduced catheter length offsets, effect of looped source transfer tubes on the average displacement, extent of inter-observer variation, and gel robustness following multiple needle-insertions. Three ProGuide sharp needles were inserted into the jar. The dwell time at each point was determined to identify the irradiated volume with a diameter of approximately 10mm on XCT images. All the times were the same. The plan was delivered using an afterloader with an Ir-192 radioactive source, and the irradiated gel dosimeter was scanned using an XCT scanner. The subtracted images were generated from pre- and post-irradiated images. Volumes with incremented Hounsfield units were manually identified and contoured. The centroid of the volume was defined as the measured source dwell position. Subsequently, planned source dwell positions were extracted from the DICOM file of the plan. Finally, the source dwell positions in plan and irradiated gel were compared in three axes. The hardness of the dosimeter was 1250% greater than that of the previously reported gel dosimeter. Source dwell positions were visually identified in the XCT image. Testing of CT acquisition, planning, irradiation, and analysis was completed in approximately 1h. In the reproducibility test of source dwell positions, created by inserting three needles (each with three source dwell positions), the average displacements of the source positions from the first source dwell position were within 0.5mm in all three directions. In the detectability test, displacements were less than 1mm in the x-y plane but greater than 1mm in the z-axis, which was the source path direction. When errors of 1-3mm were intentionally introduced, the measured displacement was within 0.7mm of the median (range: 0.21-1.65mm) of intentional errors. When the transfer tube was looped, the source dwell position displaced by approximately 1mm. After 20 needle-insertions, the source dwell position displacement was within 1mm. The maximum inter-observer variation of contouring was 0.57mm. The XCT-based gel dosimeter enabled verification of three-dimensional source dwell positions for a seamless workflow of IGBT with high precision and efficiency.

Inter-observer variation in gross tumour volume delineation of oesophageal cancer on MR, CT and PET/CT.

The aim of our study was to assess the inter-observer variability in delineation of the gross tumour volume (GTV) of oesophageal cancer on magnetic resonance (MR) in comparison to computed tomography (CT) and positron emission tomography and CT (PET/CT). Twenty-three consecutive patients with oesophageal cancer treated with chemo-radiotherapy were enrolled. All patients had PET/CT and MR imaging in treatment position. Five observers independently delineated the GTV on CT alone, MR alone, CT with co-registered MR, PET/CT alone and MR with co-registered PET/CT. Volumes of GTV were measured per patient and imaging modality. Inter-observer agreement, expressed in generalized conformity index (CIgen), volumetric conformity index (VCI), planar conformity index (PCI) and inter-delineation distance (IDD) were calculated per patient and imaging modality. Linear mixed models were used for statistical analysis. GTV volume was significantly lower on MR (33.03 cm3) compared to CT (37.1 cm3; p = 0.002) and on PET/CT MR (35.2 cm3; p = 0.018) compared to PET/CT (39.1 cm3). The CIgen was lowest on CT (0.56) and highest on PET/CT MR (0.67). The difference in CIgen between MR (0.61) and CT was borderline significant (p = 0.048). The VCI was significantly higher on MR (0.71; p = 0.007) and on CT MR (0.71; p = 0.004) compared to CT (0.67). The PCI was significantly higher on CT MR (0.67; p = 0.031) compared to CT (0.64). The largest differences were observed in the cranio-caudal direction. The highest inter-observer agreement was found for PET/CT MR and the lowest for CT. MR could reduce the difference in delineation between observers and provide additional information about the local extent of the tumour.

On-site burn severity assessment using smartphone-captured color burn wound images

Accurate assessment of burn severity is crucial for the management of burn injuries. Currently, clinicians mainly rely on visual inspection to assess burns, characterized by notable inter-observer discrepancies. In this study, we introduce an innovative analysis platform using color burn wound images for automatic burn severity assessment. To do this, we propose a novel joint-task deep learning model, which is capable of simultaneously segmenting both burn regions and body parts, the two crucial components in calculating the percentage of total body surface area (%TBSA). Asymmetric attention mechanism is introduced, allowing attention guidance from the body part segmentation task to the burn region segmentation task. A user-friendly mobile application is developed to facilitate a fast assessment of burn severity at clinical settings. The proposed framework was evaluated on a dataset comprising 1340 color burn wound images captured on-site at clinical settings. The average Dice coefficients for burn depth segmentation and body part segmentation are 85.12 % and 85.36 %, respectively. The R2 for %TBSA assessment is 0.9136. The source codes for the joint-task framework and the application are released on Github (https://github.com/xjtu-mia/BurnAnalysis). The proposed platform holds the potential to be widely used at clinical settings to facilitate a fast and precise burn assessment.

Development and comprehensive evaluation of a national DBCG consensus-based auto-segmentation model for lymph node levels in breast cancer radiotherapy

Background and purposeThis study aimed at training and validating a multi-institutional deep learning (DL) auto segmentation model for nodal clinical target volume (CTVn) in high-risk breast cancer (BC) patients with both training and validation dataset created with multi-institutional participation, with the overall aim of national clinical implementation in Denmark. Materials and methodsA gold standard (GS) dataset and a high-quality training dataset were created by 21 BC delineation experts from all radiotherapy centres in Denmark. The delineations were created according to ESTRO consensus delineation guidelines. Four models were trained: One per laterality and extension of CTVn internal mammary nodes. The DL models were tested quantitatively in their own test-set and in relation to interobserver variation (IOV) in the GS dataset with geometrical metrics, such as the Dice Similarity Coefficient (DSC). A blinded qualitative evaluation was conducted with a national board, presented to both DL and manual delineations. ResultsA median DSC > 0.7 was found for all, except the CTVn interpectoral node in one of the models. In the qualitative evaluation ‘no corrections needed’ were acquired for 297 (36 %) in the DL structures and 286 (34 %) for manual delineations. A higher rate of ‘major corrections’ and ‘easier to start from scratch’ was found in the manual delineations. The models performed within the IOV of an expert group, with two exceptions. ConclusionDL models were developed on a national consensus cohort and performed on par with the IOV between BC experts and had a comparable or higher clinical acceptance than expert manual delineations.

Development and evaluation of machine-learning methods in whole-body magnetic resonance imaging with diffusion weighted imaging for staging of patients with cancer: the MALIBO diagnostic test accuracy study

Background Whole-body magnetic resonance imaging is accurate, efficient and cost-effective for cancer staging. Machine learning may support radiologists reading whole-body magnetic resonance imaging. Objectives To develop a machine-learning algorithm to detect normal organs and cancer lesions. To compare diagnostic accuracy, time and agreement of radiology reads to detect metastases using whole-body magnetic resonance imaging with concurrent machine learning (whole-body magnetic resonance imaging + machine learning) against standard whole-body magnetic resonance imaging (whole-body magnetic resonance imaging + standard deviation). Design and participants Retrospective analysis of (1) prospective single-centre study in healthy volunteers > 18 years (n = 51) and (2) prospective multicentre STREAMLINE study patient data (n = 438). Tests Index: whole-body magnetic resonance imaging + machine learning. Comparator: whole-body magnetic resonance imaging + standard deviation. Reference standard Previously established expert panel consensus reference at 12 months from diagnosis. Outcome measures Primary: difference in per-patient specificity between whole-body magnetic resonance imaging + machine learning and whole-body magnetic resonance imaging + standard deviation. Secondary: per-patient sensitivity, per-lesion sensitivity and specificity, read time and agreement. Methods Phase 1: classification forests, convolutional neural networks, and a multi-atlas approaches for organ segmentation. Phase 2/3: whole-body magnetic resonance imaging scans were allocated to Phase 2 (training = 226, validation = 45) and Phase 3 (testing = 193). Disease sites were manually labelled. The final algorithm was applied to 193 Phase 3 cases, generating probability heatmaps. Twenty-five radiologists (18 experienced, 7 inexperienced in whole-body magnetic resonance imaging) were randomly allocated whole-body magnetic resonance imaging + machine learning or whole-body magnetic resonance imaging + standard deviation over two or three rounds in a National Health Service setting. Read time was independently recorded. Results Phases 1 and 2: convolutional neural network had best Dice similarity coefficient, recall and precision measurements for healthy organ segmentation. Final algorithm used a ‘two-stage’ initial organ identification followed by lesion detection. Phase 3: evaluable scans (188/193, of which 50 had metastases from 117 colon, 71 lung cancer cases) were read between November 2019 and March 2020. For experienced readers, per-patient specificity for detection of metastases was 86.2% (whole-body magnetic resonance imaging + machine learning) and 87.7% (whole-body magnetic resonance imaging + standard deviation), (difference −1.5%, 95% confidence interval −6.4% to 3.5%; p = 0.387); per-patient sensitivity was 66.0% (whole-body magnetic resonance imaging + machine learning) and 70.0% (whole-body magnetic resonance imaging + standard deviation) (difference −4.0%, 95% confidence interval −13.5% to 5.5%; p = 0.344). For inexperienced readers (53 reads, 15 with metastases), per-patient specificity was 76.3% in both groups with sensitivities of 73.3% (whole-body magnetic resonance imaging + machine learning) and 60.0% (whole-body magnetic resonance imaging + standard deviation). Per-site specificity remained high within all sites; above 95% (experienced) or 90% (inexperienced). Per-site sensitivity was highly variable due to low number of lesions in each site. Reading time lowered under machine learning by 6.2% (95% confidence interval −22.8% to 10.0%). Read time was primarily influenced by read round with round 2 read times reduced by 32% (95% confidence interval 20.8% to 42.8%) overall with subsequent regression analysis showing a significant effect (p = 0.0281) by using machine learning in round 2 estimated as 286 seconds (or 11%) quicker. Interobserver variance for experienced readers suggests moderate agreement, Cohen’s κ = 0.64, 95% confidence interval 0.47 to 0.81 (whole-body magnetic resonance imaging + machine learning) and Cohen’s κ = 0.66, 95% confidence interval 0.47 to 0.81 (whole-body magnetic resonance imaging + standard deviation). Limitations Patient whole-body magnetic resonance imaging data were heterogeneous with relatively few metastatic lesions in a wide variety of locations, making training and testing difficult and hampering evaluation of sensitivity. Conclusions There was no difference in diagnostic accuracy for whole-body magnetic resonance imaging radiology reads with or without machine-learning support, although radiology read time may be slightly shortened using whole-body magnetic resonance imaging + machine learning. Future work Failure-case analysis to improve model training, automate lesion segmentation and transfer of machine-learning techniques to other tumour types and imaging modalities. Study registration This study is registered as ISRCTN23068310. Funding This award was funded by the National Institute for Health and Care Research (NIHR) Efficacy and Mechanism Evaluation (EME) programme (NIHR award ref: 13/122/01) and is published in full in Efficacy and Mechanism Evaluation; Vol. 11, No. 15. See the NIHR Funding and Awards website for further award information.

Gastric-type extremely well-differentiated adenocarcinoma of the stomach: A rare tumor with diagnostic difficulties and high inter-observer variation in endoscopic pinch biopsies

Extremely well-differentiated gastric-type adenocarcinoma (EWDGA) is a rare type of gastric cancer composed of deceptively bland-looking malignant cells resembling normal foveolar or pyloric epithelium. The histological features of this tumor have not been recognized by many pathologists, and inter-observer variation studies are lacking. Here, we report seven EWDGAs and inter-observer variation of six preoperative biopsies was evaluated by 11 pathologists in a single institute. Based on the pathological diagnosis of the endoscopic biopsy slides, the average rate of definite malignancy diagnosis was 15.2 %, and the overall diagnostic concordance rate was 34.9 % among 11 pathologists. Microscopically, the surface epithelium was preserved and only a few atypical tumor glands were scattered in most endoscopic biopsies. Structural atypia was minimal, and the tumor glands were barely distinguishable from normal glands. Although nuclear atypia was minimal, enlarged nuclei, relatively large glands with irregular shapes, and abundant cytoplasmic mucin were observed in gastric pinch biopsies. In preoperative biopsies, no cases showed p53 overexpression, and Ki-67 labeling index ranged from 3 % to 35 % and was higher compared to non-neoplastic glands in 3 cases. After gastrectomy, four (57.1 %) patients had advanced gastric cancer and three (42.9 %) had lymph node metastasis. Genomic profiling of the four patients revealed mutations of TP53, BRAF, KRAS, STK11, and MDM2/CCND1 amplification. Immunohistochemistry for p53 was not helpful while Ki-67 may be helpful when staining pattern is distinct from the non-neoplastic mucosa. In conclusion, it is challenging to diagnose EWDGA using biopsy specimens. Recognizing and addressing this rare entity will increase diagnostic accuracy to ensure the early diagnosis of cancer.

Skeletal Muscle Segmentation at the Level of the Third Lumbar Vertebra (L3) in Low-Dose Computed Tomography: A Lightweight Algorithm.

The cross-sectional area of skeletal muscles at the level of the third lumbar vertebra (L3) measured from computed tomography (CT) images is an established imaging biomarker used to assess patients' nutritional status. With the increasing prevalence of low-dose CT scans in clinical practice, accurate and automated skeletal muscle segmentation at the L3 level in low-dose CT images has become an issue to address. This study proposed a lightweight algorithm for automated segmentation of skeletal muscles at the L3 level in low-dose CT images. This study included 57 patients with rectal cancer, with both low-dose plain and contrast-enhanced pelvic CT image series acquired using a radiotherapy CT scanner. A training set of 30 randomly selected patients was used to develop a lightweight segmentation algorithm, and the other 27 patients were used as the test set. A radiologist selected the most representative axial CT image at the L3 level for both the image series for all the patients, and three groups of observers manually annotated the skeletal muscles in the 54 CT images of the test set as the gold standard. The performance of the proposed algorithm was evaluated in terms of the Dice similarity coefficient (DSC), precision, recall, 95th percentile of the Hausdorff distance (HD95), and average surface distance (ASD). The running time of the proposed algorithm was recorded. An open source deep learning-based AutoMATICA algorithm was compared with the proposed algorithm. The inter-observer variations were also used as the reference. The DSC, precision, recall, HD95, ASD, and running time were 93.2 ± 1.9% (mean ± standard deviation), 96.7 ± 2.9%, 90.0 ± 2.9%, 4.8 ± 1.3 mm, 0.8 ± 0.2 mm, and 303 ± 43 ms (on CPU) for the proposed algorithm, and 94.1 ± 4.1%, 92.7 ± 5.5%, 95.7 ± 4.0%, 7.4 ± 5.7 mm, 0.9 ± 0.6 mm, and 448 ± 40 ms (on GPU) for AutoMATICA, respectively. The differences between the proposed algorithm and the inter-observer reference were 4.7%, 1.2%, 7.9%, 3.2 mm, and 0.6 mm, respectively, for the averaged DSC, precision, recall, HD95, and ASD. The proposed algorithm can be used to segment skeletal muscles at the L3 level in either the plain or enhanced low-dose CT images.

Comprehensive Clinical Usability-Oriented Contour Quality Evaluation for Deep Learning Auto-segmentation: Combining Multiple Quantitative Metrics Through Machine Learning

PurposeThe current commonly-used metrics for evaluating the quality of auto-segmented contours have limitations and do not always reflect the clinical usefulness of the contours. This work aims to develop a novel contour quality classification (CQC) method by combining multiple quantitative metrics for clinical usability-oriented contour quality evaluation for deep learning-based auto-segmentation (DLAS). MethodsThe CQC was designed to categorize contours on slices as acceptable, minor edit, or major edit based on the expected editing effort/time with supervised ensemble tree classification models using seven quantitative metrics. Organ-specific models were trained for five abdominal organs (pancreas, duodenum, stomach, small and large-bowels) using 50 MRI datasets. Twenty additional MRI and nine CT datasets were employed for testing. Inter-observer variation (IOV) was assessed among six observers and consensus labels were established through majority vote for evaluation. The CQC was also compared with a threshold-based baseline approach. ResultsFor the five organs, the average AUC was 0.982±0.01 and 0.979±0.01, the mean-accuracy was 95.8±1.7% and 94.3±2.1%, and the mean risk-rate was 0.8±0.4% and 0.7±0.5% for MRI and CT testing dataset, respectively. The CQC results closely matched the IOV results (mean-accuracy of 94.2±0.8% and 94.8±1.7%) and were significantly higher than those obtained using the threshold-based method (mean-accuracy of 80.0±4.7%, 83.8±5.2%, and 77.3±6.6% using one, two, and three metrics). ConclusionThe CQC models demonstrated high performance in classifying the quality of contour slices. This method can address the limitations of existing metrics and offers an intuitive and comprehensive solution for clinically oriented evaluation and comparison of DLAS systems.

Association of Oncotype-DX HER2 Single Gene Score With HER2 Expression Assessed by Immunohistochemistry in HER2-low Breast Cancer.

"HER2-low" is an emerging subtype of breast cancer, with a documented role in predicting response to treatment with novel antibody-drug conjugates. It is defined based on immunohistochemistry, but increasing evidence is challenging this approach as appropriate for identifying the HER2-low subgroup, due to both interobserver variability and limitations of the method itself. We retrospectively analyzed data from 430 patients from our departmental databases who had been subjected to an Oncotype-DX score and assessed the correlation of the Oncotype-DX HER2 single-gene score with the HER2 expression on immunohistochemistry. The Oncotype-DX Recurrence Score was also evaluated in the HER2-0 versus HER2-low subgroups. The HER2 single-gene score was found to accurately correlate with the HER2 result on immunohistochemistry, with a statistically significant difference both between HER2-0 and HER2 +1 tumors (p<0.0001), as well as between HER2 +1 and +2 tumors (p<0.0001). There was no statistically significant difference in the recurrence score between the HER2-0 and the HER2-low subgroups. Oncotype-DX single-gene scores for HER2 are a potential surrogate marker for assessing the precise HER2 status, with better reproducibility and less interobserver variance compared to immunohistochemistry. The use of rt-PCR emerges as an alternative method of assessment of the HER2-low subgroup.

2204: Motion management strategy and interobserver variation in abdominal OAR delineation using MRI

2204: Motion management strategy and interobserver variation in abdominal OAR delineation using MRI

Deep learning with uncertainty estimation for automatic tumor segmentation in PET/CT of head and neck cancers: impact of model complexity, image processing and augmentation

Objective. Target volumes for radiotherapy are usually contoured manually, which can be time-consuming and prone to inter- and intra-observer variability. Automatic contouring by convolutional neural networks (CNN) can be fast and consistent but may produce unrealistic contours or miss relevant structures. We evaluate approaches for increasing the quality and assessing the uncertainty of CNN-generated contours of head and neck cancers with PET/CT as input. Approach. Two patient cohorts with head and neck squamous cell carcinoma and baseline 18F-fluorodeoxyglucose positron emission tomography and computed tomography images (FDG-PET/CT) were collected retrospectively from two centers. The union of manual contours of the gross primary tumor and involved nodes was used to train CNN models for generating automatic contours. The impact of image preprocessing, image augmentation, transfer learning and CNN complexity, architecture, and dimension (2D or 3D) on model performance and generalizability across centers was evaluated. A Monte Carlo dropout technique was used to quantify and visualize the uncertainty of the automatic contours. Main results. CNN models provided contours with good overlap with the manually contoured ground truth (median Dice Similarity Coefficient: 0.75–0.77), consistent with reported inter-observer variations and previous auto-contouring studies. Image augmentation and model dimension, rather than model complexity, architecture, or advanced image preprocessing, had the largest impact on model performance and cross-center generalizability. Transfer learning on a limited number of patients from a separate center increased model generalizability without decreasing model performance on the original training cohort. High model uncertainty was associated with false positive and false negative voxels as well as low Dice coefficients. Significance. High quality automatic contours can be obtained using deep learning architectures that are not overly complex. Uncertainty estimation of the predicted contours shows potential for highlighting regions of the contour requiring manual revision or flagging segmentations requiring manual inspection and intervention.

Factors influencing the reliability of measurements in eyes with full-thickness macular holes: are we measuring incorrectly?

PurposeThe calliper function is used for manual measurements of full thickness macular holes (FTMHs). We aimed to investigate whether a reproducible difference could be detected beyond interobserver variability between two...

Agreement of Pain Assessment Using the Short Form of the Canine Glasgow Composite Measure Pain Scale between Veterinary Students, Veterinary Nurses, Veterinary Surgeons, and ECVAA-Diplomates.

Several pain scoring systems have been validated to measure pain in dogs. However, pain may not be adequately assessed since these tools are associated with high-level inter-observer variation. The aim of this study is to evaluate the agreement of pain assessment using the CMPS-SF between veterinary students, veterinary nurses, veterinary surgeons, and European College of Veterinary Anaesthesia and Analgesia (ECVAA) diplomates. Forty-five client-owned dogs presented to a teaching hospital were enrolled in this prospective, observational study. All dogs were pain-scored in vivo, while a video of the assessment was recorded and subsequently evaluated by twenty assessors, with five per group. Mean scores between groups were compared, and agreement within groups and agreement of the average scores between groups were assessed by calculating the intraclass correlation coefficient (ICC). The intervention point at which dogs were deemed to require additional analgesia was also evaluated. Overall agreement of pain assessment was poor (ICC = 0.494). Nurses had the best inter-observer agreement (ICC = 0.656), followed by ECVAA diplomates (ICC = 0.540), veterinary surgeons (ICC = 0.478), and veterinary students (ICC = 0.432). The best inter-group agreement was between veterinary surgeons and nurses (ICC = 0.951) and between ECVAA diplomates and nurses (ICC = 0.951). Students were more likely to determine that additional analgesia was required compared to other groups. Pain assessment is key for animal welfare, and training in this area should be reinforced to improve consistency.

Challenges with segmenting intraoperative ultrasound for brain tumours

Objective - Addressing the challenges that come with identifying and delineating brain tumours in intraoperative ultrasound. Our goal is to both qualitatively and quantitatively assess the interobserver variation, amongst experienced neuro-oncological intraoperative ultrasound users (neurosurgeons and neuroradiologists), in detecting and segmenting brain tumours on ultrasound. We then propose that, due to the inherent challenges of this task, annotation by localisation of the entire tumour mass with a bounding box could serve as an ancillary solution to segmentation for clinical training, encompassing margin uncertainty and the curation of large datasets. Methods - 30 ultrasound images of brain lesions in 30 patients were annotated by 4 annotators - 1 neuroradiologist and 3 neurosurgeons. The annotation variation of the 3 neurosurgeons was first measured, and then the annotations of each neurosurgeon were individually compared to the neuroradiologist’s, which served as a reference standard as their segmentations were further refined by cross-reference to the preoperative magnetic resonance imaging (MRI). The following statistical metrics were used: Intersection Over Union (IoU), Sørensen-Dice Similarity Coefficient (DSC) and Hausdorff Distance (HD). These annotations were then converted into bounding boxes for the same evaluation. Results - There was a moderate level of interobserver variance between the neurosurgeons [IoU:0.789,DSC:0.876,HD:103.227]\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$[IoU:0.789, \\ DSC:0.876, \\ HD:103.227]$$\\end{document} and a larger level of variance when compared against the MRI-informed reference standard annotations by the neuroradiologist, mean across annotators [IoU:0.723,DSC:0.813,HD:115.675]\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$[IoU:0.723, \\ DSC:0.813, \\ HD:115.675]$$\\end{document}. After converting the segments to bounding boxes, all metrics improve, most significantly, the interquartile range drops by [IoU:37%,DSC:41%,HD:54%]\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$[IoU:37\\%, \\ DSC:41\\%, \\ HD:54\\%]$$\\end{document}. Conclusion - This study highlights the current challenges with detecting and defining tumour boundaries in neuro-oncological intraoperative brain ultrasound. We then show that bounding box annotation could serve as a useful complementary approach for both clinical and technical reasons.

Learning three-dimensional aortic root assessment based on sparse annotations.

Analyzing the anatomy of the aorta and left ventricular outflow tract (LVOT) is crucial for risk assessment and planning of transcatheter aortic valve implantation (TAVI). A comprehensive analysis of the aortic root and LVOT requires the extraction of the patient-individual anatomy via segmentation. Deep learning has shown good performance on various segmentation tasks. If this is formulated as a supervised problem, large amounts of annotated data are required for training. Therefore, minimizing the annotation complexity is desirable. We propose two-dimensional (2D) cross-sectional annotation and point cloud-based surface reconstruction to train a fully automatic 3D segmentation network for the aortic root and the LVOT. Our sparse annotation scheme enables easy and fast training data generation for tubular structures such as the aortic root. From the segmentation results, we derive clinically relevant parameters for TAVI planning. The proposed 2D cross-sectional annotation results in high inter-observer agreement [Dice similarity coefficient (DSC): 0.94]. The segmentation model achieves a DSC of 0.90 and an average surface distance of 0.96mm. Our approach achieves an aortic annulus maximum diameter difference between prediction and annotation of 0.45mm (inter-observer variance: 0.25mm). The presented approach facilitates reproducible annotations. The annotations allow for training accurate segmentation models of the aortic root and LVOT. The segmentation results facilitate reproducible and quantifiable measurements for TAVI planning.

Inter- and intra-observer variation in phytolith morphometry.

Archaeobotanists and palaeoecologists extensively use geometric morphometrics to identify plant opal phytoliths. Particularly when applied to assemblages of phytoliths from concentrations retrieved from closed contexts, morphometric data from archaeological phytoliths compared with similar data from reference material may allow taxonomic attribution. Observer variation is one aspect of phytolith morphometry that has received little attention but may be an important source of error, and hence cause of potential misidentification of plant remains. To investigate inter- and intra-observer variation in phytolith morphometry, eight researchers (observers) from different laboratories measured 50 samples each from three phytolith morphotypes, Bilobate, Bulliform flabellate and Elongate dendritic, three times, under the auspices of the International Committee for Phytolith Morphometrics (ICPM). Data for 17 size and shape variables were collected for each phytolith by manually digitising a phytolith outline (mask) from a photograph, followed by measurement of the mask with open-source morphometric software. Inter-observer variation ranged from 0 to 23% difference from the mean of all observers. Intra-observer variation ranged from 0 to 9% difference from the mean of individual observers per week. Inter- and intra-observer variation was generally higher among inexperienced researchers. Scaling errors were a major cause of variation and occurred more with less experienced researchers, which is likely related to familiarity with data collection. The results indicate that inter- and intra-observer variation can be substantially reduced by providing clear instructions for and training with the equipment, photo capturing, software, data collection and data cleaning. In this paper, the ICPM provides recommendations to minimise variation.Advances in automatic data collection may eventually reduce inter- and intra-observer variation, but until this is common practice, the ICPM recommends that phytolith morphometric analyses adhere to standardised guidelines to assure that measured phytolith variables are accurate, consistent and comparable between different researchers and laboratories.