Simultaneous Truth And Performance Level Estimation Algorithm Research Articles

Validation of QyScore’s® fully automated quantitative image segmentation tools against expert manual gold‐standards

AbstractBackgroundQuantitative imaging provides valuable information for early detection, disease progression and treatment response monitoring. Manual segmentation is the current gold‐standard, but is prohibitively labour‐intensive for large scale use, such as in clinical routine, and suffers from individual variability. Thus, there is a substantial unmet need for validation of automatic segmentation techniques that perform as accurately as this labour‐intensive manual gold‐standard.MethodThe validation cohort consisted of 50 individuals with multiple diagnoses, wide age range (18‐86 yrs), balanced for sex and acquired on multiple scanners for improved generalizability of quantification results (Table 1). Three expert neuroradiologists each manually segmented the caudate, putamen, globus pallidus, thalamus, cerebellum, brainstem, and lateral ventricles on 3DT1 images using itk‐SNAP (http://www.itksnap.org/) software. A consensus expert segmentation was then derived using the STAPLE (Simultaneous Truth and Performance Level Estimation) algorithm (Wakefield et al., (2004)) for each region and compared with the fully automated segmentations produced using QyScore®, a CE‐marked and FDA‐cleared neuroimaging medical device. Performance was investigated using the Dice Similarity Coefficient (DSC) and concordance assessed with plotted linear regression.ResultMean, standard deviation and confidence intervals of the DSC demonstrated a high degree of agreement between the consensus manual gold‐standard and QyScore®’s automated segmentations (Table 2). Importantly, this agreement remained consistent following stratification by field strength, demonstrating generalizability to most clinical imaging centres. For 1.5T scanners the mean DSC was 0.81, 0.89, 0.79, 0.86, 0.94, 0.93 and 0.91 for the caudate, putamen, globus pallidus, thalamus, cerebellum, brainstem, and lateral ventricles respectively. For the 3T scanners, this was equivalent or marginally better at 0.84, 0.89, 0.82, 0.86, 0.94, 0.94, and 0.93 respectively. In addition, strong concordance was demonstrated between the volumes, expressed as a percentage of intracranial volume (%ICV), obtained by the automated medical device QyScore® and the consensus of the manual gold‐standard segmentation (Figure 1: Table 2).ConclusionQyScore® produces fast reliable automated segmentations with comparable accuracy to expert neuroradiologists. These findings support the implementation of QyScore® in clinical trials and in clinical routine to provide quantitative image analysis in support of diagnosis and monitoring of disease progression and treatment response.

Validation of QyScore’s® fully automated quantitative image segmentation tools against expert manual gold‐standards

AbstractBackgroundQuantitative imaging provides valuable information for early detection, disease progression and treatment response monitoring. Manual segmentation is the current gold‐standard, but is prohibitively labour‐intensive for large scale use, such as in clinical routine, and suffers from individual variability. Thus, there is a substantial unmet need for validation of automatic segmentation techniques that perform as accurately as this labour‐intensive manual gold‐standard.MethodThe validation cohort consisted of 50 individuals with multiple diagnoses, wide age range (18‐86 yrs), balanced for sex and acquired on multiple scanners for improved generalizability of quantification results (Table 1). Three expert neuroradiologists each manually segmented the caudate, putamen, globus pallidus, thalamus, cerebellum, brainstem, and lateral ventricles on 3DT1 images using itk‐SNAP (http://www.itksnap.org/) software. A consensus expert segmentation was then derived using the STAPLE (Simultaneous Truth and Performance Level Estimation) algorithm (Wakefield et al., (2004)) for each region and compared with the fully automated segmentations produced using QyScore®, a CE‐marked and FDA‐cleared neuroimaging medical device. Performance was investigated using the Dice Similarity Coefficient (DSC) and concordance assessed with plotted linear regression.ResultMean, standard deviation and confidence intervals of the DSC demonstrated a high degree of agreement between the consensus manual gold‐standard and QyScore®’s automated segmentations (Table 2). Importantly, this agreement remained consistent following stratification by field strength, demonstrating generalizability to most clinical imaging centres. For 1.5T scanners the mean DSC was 0.81, 0.89, 0.79, 0.86, 0.94, 0.93 and 0.91 for the caudate, putamen, globus pallidus, thalamus, cerebellum, brainstem, and lateral ventricles respectively. For the 3T scanners, this was equivalent or marginally better at 0.84, 0.89, 0.82, 0.86, 0.94, 0.94, and 0.93 respectively. In addition, strong concordance was demonstrated between the volumes, expressed as a percentage of intracranial volume (%ICV), obtained by the automated medical device QyScore® and the consensus of the manual gold‐standard segmentation (Figure 1: Table 2).ConclusionQyScore® produces fast reliable automated segmentations with comparable accuracy to expert neuroradiologists. These findings support the implementation of QyScore® in clinical trials and in clinical routine to provide quantitative image analysis in support of diagnosis and monitoring of disease progression and treatment response.

Chákṣu: A glaucoma specific fundus image database

We introduce Chákṣu–a retinal fundus image database for the evaluation of computer-assisted glaucoma prescreening techniques. The database contains 1345 color fundus images acquired using three brands of commercially available fundus cameras. Each image is provided with the outlines for the optic disc (OD) and optic cup (OC) using smooth closed contours and a decision of normal versus glaucomatous by five expert ophthalmologists. In addition, segmentation ground-truths of the OD and OC are provided by fusing the expert annotations using the mean, median, majority, and Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm. The performance indices show that the ground-truth agreement with the experts is the best with STAPLE algorithm, followed by majority, median, and mean. The vertical, horizontal, and area cup-to-disc ratios are provided based on the expert annotations. Image-wise glaucoma decisions are also provided based on majority voting among the experts. Chákṣu is the largest Indian-ethnicity-specific fundus image database with expert annotations and would aid in the development of artificial intelligence based glaucoma diagnostics.

Localization of cementoenamel junction in intraoral ultrasonographs with machine learning

ObjectiveOur goal was to automatically identify the cementoenamel junction (CEJ) location in ultrasound images using deep convolution neural networks (CNNs). MethodsThree CNNs were evaluated using 1400 images and data augmentation. The training and validation were performed by an experienced nonclinical rater with 1000 and 200 images, respectively. Four clinical raters with different levels of experience with ultrasound tested the networks using the other 200 images. In addition to the comparison of the best approach with each rater, we also employed the simultaneous truth and performance level estimation (STAPLE) algorithm to estimate a ground truth based on all labelings by four clinical raters. The final CEJ location estimate was obtained by taking the first moment of the posterior probability computed using the STAPLE algorithm. The study also computed the machine learning-measured CEJ–alveolar bone crest distance. ResultsQuantitative evaluations of the 200 images showed that the comparison of the best approach with the STAPLE-estimate yielded a mean difference (MD) of 0.26 mm, which is close to the comparison with the most experienced nonclinical rater (MD=0.25 mm) but far better than the comparison with clinical raters (MD=0.27–0.33 mm). The machine learning-measured CEJ–alveolar bone crest distances correlated strongly (R = 0.933, p < 0.001) with the manual clinical labeling and the measurements were in good agreement with the 95% Bland–Altman's lines of agreement between −0.68 and 0.57 mm. ConclusionsThe study demonstrated the feasible use of machine learning methodology to localize CEJ in ultrasound images with clinically acceptable accuracy and reliability. Likelihood-weighted ground truth by combining multiple labels by the clinical experts compared favorably with the predictions by the best deep CNN approach. Clinical significanceIdentification of CEJ and its distance from the alveolar bone crest play an important role in the evaluation of periodontal status. Machine learning algorithms can learn from complex features in ultrasound images and have potential to provide a reliable and accurate identification in subsecond. This will greatly assist dental practitioners to provide better point-of-care to patients and enhance the throughput of dental care.

Esophagus Segmentation in CT Images via Spatial Attention Network and STAPLE Algorithm.

One essential step in radiotherapy treatment planning is the organ at risk of segmentation in Computed Tomography (CT). Many recent studies have focused on several organs such as the lung, heart, esophagus, trachea, liver, aorta, kidney, and prostate. However, among the above organs, the esophagus is one of the most difficult organs to segment because of its small size, ambiguous boundary, and very low contrast in CT images. To address these challenges, we propose a fully automated framework for the esophagus segmentation from CT images. The proposed method is based on the processing of slice images from the original three-dimensional (3D) image so that our method does not require large computational resources. We employ the spatial attention mechanism with the atrous spatial pyramid pooling module to locate the esophagus effectively, which enhances the segmentation performance. To optimize our model, we use group normalization because the computation is independent of batch sizes, and its performance is stable. We also used the simultaneous truth and performance level estimation (STAPLE) algorithm to reach robust results for segmentation. Firstly, our model was trained by k-fold cross-validation. And then, the candidate labels generated by each fold were combined by using the STAPLE algorithm. And as a result, Dice and Hausdorff Distance scores have an improvement when applying this algorithm to our segmentation results. Our method was evaluated on SegTHOR and StructSeg 2019 datasets, and the experiment shows that our method outperforms the state-of-the-art methods in esophagus segmentation. Our approach shows a promising result in esophagus segmentation, which is still challenging in medical analyses.

Large Interobserver Variation in the International MR-LINAC Oropharyngeal Carcinoma Delineation Study

Generally, a (contrast-enhanced) CT is used for delineation of oropharyngeal carcinomas (OPC’s). MRI however, has a superior soft tissue contrast. Adding MR images to the planning CT, might result in a reduced interobserver variation. MR-guided radiotherapy is in its infancy but will be the standard in the near future. Therefore, we want to determine the variation in target delineation for oropharyngeal cancer (OPC) by international experts using MR as a single delineation modality in order to develop guidelines for delineating OPC’s on MR images Twentyfour radiation oncologists from 7 centers affiliated with the MR-linac Consortium were asked to delineate the gross tumor volume (GTV) and the clinical target volume (CTV) of the primary tumor for 4 OPC cases. Observers were given a brief clinical history, a physical examination report and the T1-, T1- plus Gadolinium and T2- weighted MR images . Target contours were delineated according to experts’ institutional guidelines. The absolute and encompassing volumes, the generalized conformity index and median surface distances were then calculated from the voxel count maps of each case. Furthermore, the data were analyzed with the Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm. The median years of experience for the participating radiation oncologists was 5 years (range 0-15 years). The mean absolute volumes of the GTVs of case 1, case 2, case 3 and case 4 were 6.5 ml (range: 3.9-9.9 ml), 8.6 ml (range 1.9-22.1 ml), 32.6 ml (range: 17.3-64.4 ml) and 20.3 ml (range 7.8-33.6 ml), respectively. The conformity indices for the GTV of the primary tumor were 0.6, 0.38, 0.5 and 0.59 for case 1, case 2, case 3 and case 4, respectively. The mean absolute volumes of the CTVs of case 1, case 2, case 3 and case 4 were 37.0 ml (range: 9.9 - 76.6 ml), 41.4 ml (range 19.7 - 108.5 ml), 113.8 ml (range: 48.5 - 254.7 ml) and 77.4 ml (range 22.7 - 140.6 ml), respectively. The conformity indices for the CTV of the primary tumor were 0.52, 0.46, 0.49 and 0.56 for case 1, case 2, case 3 and case 4 respectively. The STAPLE algorithm largely covers the median GTV voxel density. The interobserver variation of target volumes delineation of OPC patient based on MR is large. For the GTV, volume differences up to a factor of 10 were observed. For the CTV, differences up to of factor of 8 were observed.This demonstrates the need for international MR delineation guidelines for oropharyngeal cancer patients treated using MR-guided radiotherapy.

Atlas-based segmentation in breast cancer radiotherapy: Evaluation of specific and generic-purpose atlases

ObjectivesAtlas-based automatic segmentation (ABAS) addresses the challenges of accuracy and reliability in manual segmentation. We aim to evaluate the contribution of specific-purpose in ABAS of breast cancer (BC) patients with respect to generic-purpose libraries. Materials and methodsOne generic-purpose and 9 specific-purpose libraries, stratified according to type of surgery and size of thorax circumference, were obtained from the computed tomography of 200 BC patients. Keywords about contralateral breast volume and presence of breast expander/prostheses were recorded. ABAS was validated on 47 independent patients, considering manual segmentation from scratch as reference. Five ABAS datasets were obtained, testing single-ABAS and multi-ABAS with simultaneous truth and performance level estimation (STAPLE). Center of mass distance (CMD), average Hausdorff distance (AHD) and Dice similarity coefficient (DSC) between corresponding ABAS and manual structures were evaluated and statistically significant differences between different surgeries, structures and ABAS strategies were investigated. ResultsStatistically significant differences between patients who underwent different surgery were found, with superior results for conservative-surgery group, and between different structures were observed: ABAS of heart, lungs, kidneys and liver was satisfactory (median values: CMD<2 mm, DSC≥0.80, AHD<1.5 mm), whereas chest wall, breast and spinal cord obtained moderate performance (median values: 2 mm ≤ CMD<5 mm, 0.60 ≤ DSC<0.80, 1.5 mm ≤ AHD<4 mm) and esophagus, stomach, brachial plexus and supraclavicular nodes obtained poor performance (median CMD≥5 mm, DSC<0.60, AHD≥4 mm). The application of STAPLE algorithm generally yields higher performance and the use of keywords improves results for breast ABAS. ConclusionThe homogeneity in the selection of atlases based on multiple anatomical and clinical features and the use of specific-purpose libraries can improve ABAS performance with respect to generic-purpose libraries.

Is STAPLE algorithm confident to assess segmentation methods in PET imaging?

Accurate tumor segmentation in [18F]-fluorodeoxyglucose positron emission tomography is crucial for tumor response assessment and target volume definition in radiation therapy. Evaluation of segmentation methods from clinical data without ground truth is usually based on physicians’ manual delineations. In this context, the simultaneous truth and performance level estimation (STAPLE) algorithm could be useful to manage the multi-observers variability. In this paper, we evaluated how this algorithm could accurately estimate the ground truth in PET imaging.Complete evaluation study using different criteria was performed on simulated data. The STAPLE algorithm was applied to manual and automatic segmentation results. A specific configuration of the implementation provided by the Computational Radiology Laboratory was used.Consensus obtained by the STAPLE algorithm from manual delineations appeared to be more accurate than manual delineations themselves (80% of overlap). An improvement of the accuracy was also observed when applying the STAPLE algorithm to automatic segmentations results.The STAPLE algorithm, with the configuration used in this paper, is more appropriate than manual delineations alone or automatic segmentations results alone to estimate the ground truth in PET imaging. Therefore, it might be preferred to assess the accuracy of tumor segmentation methods in PET imaging.

Dosimetric and Radiobiological Consequences of Computed Tomography–Guided Adaptive Strategies for Intensity Modulated Radiation Therapy of the Prostate

To examine a range of scenarios for image-guided adaptive radiation therapy of prostate cancer, including different schedules for megavoltage CT imaging, patient repositioning, and dose replanning. We simulated multifraction dose distributions with deformable registration using 35 sets of megavoltage CT scans of 13 patients. We computed cumulative dose-volume histograms, from which tumor control probabilities and normal tissue complication probabilities (NTCPs) for rectum were calculated. Five-field intensity modulated radiation therapy (IMRT) with 18-MV x-rays was planned to achieve an isocentric dose of 76 Gy to the clinical target volume (CTV). The differences between D95, tumor control probability, V70Gy, and NTCP for rectum, for accumulated versus planned dose distributions, were compared for different target volume sizes, margins, and adaptive strategies. The CTV D95 for IMRT treatment plans, averaged over 13 patients, was 75.2 Gy. Using the largest CTV margins (10/7 mm), the D95 values accumulated over 35 fractions were within 2% of the planned value, regardless of the adaptive strategy used. For tighter margins (5 mm), the average D95 values dropped to approximately 73.0 Gy even with frequent repositioning, and daily replanning was necessary to correct this deficit. When personalized margins were applied to an adaptive CTV derived from the first 6 treatment fractions using the STAPLE (Simultaneous Truth and Performance Level Estimation) algorithm, target coverage could be maintained using a single replan 1 week into therapy. For all approaches, normal tissue parameters (rectum V(70Gy) and NTCP) remained within acceptable limits. The frequency of adaptive interventions depends on the size of the CTV combined with target margins used during IMRT optimization. The application of adaptive target margins (<5 mm) to an adaptive CTV determined 1 week into therapy minimizes the need for subsequent dose replanning.

Agreement Among RTOG Sarcoma Radiation Oncologists in Contouring Suspicious Peritumoral Edema for Preoperative Radiation Therapy of Soft Tissue Sarcoma of the Extremity

Peritumoral edema may harbor sarcoma cells. The extent of suspicious edema (SE) included in the treatment volume is subject to clinical judgment, balancing the risk of missing tumor cells with excess toxicity. Our goal was to determine variability in SE delineation by sarcoma radiation oncologists (RO). Twelve expert ROs were provided with T1 gadolinium and T2-weighted MR images of 10 patients with high-grade extremity soft-tissue sarcoma. Gross tumor volume, clinical target volume (CTV)3cm (3 cm longitudinal and 1.5 cm radial margin), and CTV2cm (2 cm longitudinal and 1 cm radial margin) were contoured by a single observer. Suspicious peritumoral edema, defined as abnormal signal on T2 images, was independently delineated by all 12 ROs. Contouring agreement was analyzed using the simultaneous truth and performance level estimation (STAPLE) algorithm and kappa statistics. The mean volumes of GTV, CTV2cm, and CTV3cm were, respectively, 130 cm(3) (7-413 cm(3)), 280 cm(3) and 360 cm(3). The mean consensus volume computed using the STAPLE algorithm at 95% confidence interval was 188 cm(3) (24-565 cm(3)) with a substantial overall agreement corrected for chance (mean kappa = 0.71; range: 0.32-0.87). The minimum, maximum, and mean volume of SE (excluding the GTV) were 4, 182, and 58 cm(3) (representing a median of 29% of the GTV volume). The median volume of SE not included in the CTV2cm and in the CTV3cm was 5 and 0.3 cm(3), respectively. There were 3 large tumors with >30 cm(3) of SE not included in the CTV3cm volume. Despite the fact that SE would empirically seem to be a more subjective volume, a substantial or near-perfect interobserver agreement was observed in SE delineation in most cases with high-grade soft-tissue sarcomas of the extremity. A median of 97% of the consensus SE is within the CTV2cm (99.8% within the CTV3cm). In a minority of cases, however, significant expansion of the CTVs is required to cover SE.

Combining prior day contours to improve automated prostate segmentation

To improve the accuracy of automatically segmented prostate, rectum, and bladder contours required for online adaptive therapy. The contouring accuracy on the current image guidance [image guided radiation therapy (IGRT)] scan is improved by combining contours from earlier IGRT scans via the simultaneous truth and performance level estimation (STAPLE) algorithm. Six IGRT prostate patients treated with daily kilo-voltage (kV) cone-beam CT (CBCT) had their original plan CT and nine CBCTs contoured by the same physician. Three types of automated contours were produced for analysis. (1) Plan: By deformably registering the plan CT to each CBCT and then using the resulting deformation field to morph the plan contours to match the CBCT anatomy. (2) Previous: The contour set drawn by the physician on the previous day CBCT is similarly deformed to match the current CBCT anatomy. (3) STAPLE: The contours drawn by the physician, on each prior CBCT and the plan CT, are deformed to match the CBCT anatomy to produce multiple contour sets. These sets are combined using the STAPLE algorithm into one optimal set. Compared to plan and previous, STAPLE improved the average Dice's coefficient (DC) with the original physician drawn CBCT contours to a DC as follows: Bladder: 0.81 ± 0.13, 0.91 ± 0.06, and 0.92 ± 0.06; Prostate: 0.75 ± 0.08, 0.82 ± 0.05, and 0.84 ± 0.05; and Rectum: 0.79 ± 0.06, 0.81 ± 0.06, and 0.85 ± 0.04, respectively. The STAPLE results are within intraobserver consistency, determined by the physician blindly recontouring a subset of CBCTs. Comparing plans recalculated using the physician and STAPLE contours showed an average disagreement less than 1% for prostate D98 and mean dose, and 5% and 3% for bladder and rectum mean dose, respectively. One scan takes an average of 19 s to contour. Using five scans plus STAPLE takes less than 110 s on a 288 core graphics processor unit. Combining the plan and all prior days via the STAPLE algorithm to produce treatment day contours is superior to the current standard of deforming only the plan contours to the daily CBCT. STAPLE also improves the precision, with a substantial decrease in standard deviation, a key for adaptive therapy. Geometrically and dosimetrically accurate contours can be automatically generated with STAPLE on prostate region kV CBCT in a time scale suitable for online adaptive therapy.

WE-E-213CD-09: Multi-Atlas Fusion Using a Tissue Appearance Model.

To improve multi-atlas based auto-segmentation by integrating a tissue appearance model with the Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm to perform multi-atlas fusion. Ten head-and-neck planning CT images were acquired (resolution: 1.0×1.0×2.5mm3 ) and the parotid glands were contoured manually by a head-and-neck oncologist. We performed 10 leave-one-out tests by using one patient as test patient and the rest of 9 patients as atlases. Deformable registration was first applied to transform the atlas parotid contours to the test image one by one. The STAPLE algorithm was initialized by a parotid tissue appearance model, which was estimated from the test image and encoded the intensity information of parotid glands. The individual deformed contours were then fused using the STAPLE algorithm to produce a best approximation of the true contour. The tissue appearance model was also applied to a deformable model segmentation to further refine the fused contours. The multi-atlas fusion using the tissue appearance model produced an average Dice coefficient of 85.2%±3.1% (left parotid) and 84.9%±3.9% (right parotid) over the 10 tests between the auto-contour and the manual contour, and an average mean surface distance of 1.6±0.3mm and 1.6±0.4mm for left and right parotids respectively. This demonstrated a good agreement between the manual contours and the auto- delineated contours. Our results also showed that, without using the tissue appearance model, the auto-delineated parotid contours might include nearby bony structures; however, using the appearance model was able to correct this problem. Including the intensity information using a tissue appearance model into STAPLE algorithm for multi-atlas fusion showed improvement in refining the anatomical boundaries in the multi- atlas based auto-segmentation.

SU‐E‐J‐99: Multi‐Atlas Based Auto‐Segmentation of Low‐Risk Clinical Target Volume (CTV) for Head‐and‐Neck Radiotherapy

Purpose: To develop an auto‐segmentation approach using multiple atlases to improve the quality of target delineation for low‐risk CTV of unilateral tonsil cancers. Methods: Sixteen unilateral tonsil cancer patients received IMRT treatment for their left tonsil tumors. These patients were treated by a total of 8 oncologists, who delineated all contours manually on the planning CT. We chose 6 patients as atlases, and used deformable image registration to map each atlas CTV to the rest of 10 test patients. In each test patient, the final contour was produced by fusing 6 individually deformed contours using the Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm. In addition, we also identified the best single atlas which best matched physicianˈs contour on each test case. The auto‐segmented contours were evaluated by comparing against the physician delineated contours using slice‐wise Hausdorff distance (HD) and Dice coefficient (DSC), and a total volume overlap index. Results: Evaluation results showed that not a single atlas could produce reasonably good results in all 10 testing cases. The multi‐atlas method achieved much better agreement than the single best atlas method. The multi‐atlas method produced the median slice‐wise HD of 7.35mm, the median slice‐wise DSC of 80.2%, and the total volume overlap of 77.8%. For radiation oncologists who contoured both the test case and one of the atlas cases, the best atlas almost always came from the same radiation oncologist, indicating that inter‐observer variation is dominated in target delineation. Our multi‐atlas result confirms that the use of multi‐atlas and STAPLE algorithm automatically reduces the inter‐observer variability. Conclusions: We developed and validated a multi‐atlas based auto‐segmentation method to improve the quality and usability of low‐risk target volume delineation for unilateral tonsil cancer radiotherapy.

Evaluation of Automatic Atlas-Based Lymph Node Segmentation for Head-and-Neck Cancer

To evaluate if automatic atlas-based lymph node segmentation (LNS) improves efficiency and decreases inter-observer variability while maintaining accuracy. Five physicians with head-and-neck IMRT experience used computed tomography (CT) data from 5 patients to create bilateral neck clinical target volumes covering specified nodal levels. A second contour set was automatically generated using a commercially available atlas. Physicians modified the automatic contours to make them acceptable for treatment planning. To assess contour variability, the Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm was used to take collections of contours and calculate a probabilistic estimate of the "true" segmentation. Differences between the manual, automatic, and automatic-modified (AM) contours were analyzed using multiple metrics. Compared with the "true" segmentation created from manual contours, the automatic contours had a high degree of accuracy, with sensitivity, Dice similarity coefficient, and mean/max surface disagreement values comparable to the average manual contour (86%, 76%, 3.3/17.4 mm automatic vs. 73%, 79%, 2.8/17 mm manual). The AM group was more consistent than the manual group for multiple metrics, most notably reducing the range of contour volume (106-430 mL manual vs. 176-347 mL AM) and percent false positivity (1-37% manual vs. 1-7% AM). Average contouring time savings with the automatic segmentation was 11.5 min per patient, a 35% reduction. Using the STAPLE algorithm to generate "true" contours from multiple physician contours, we demonstrated that, in comparison with manual segmentation, atlas-based automatic LNS for head-and-neck cancer is accurate, efficient, and reduces interobserver variability.

Estimation of Inferential Uncertainty in Assessing Expert Segmentation Performance From STAPLE

The evaluation of the quality of segmentations of an image, and the assessment of intra- and inter-expert variability in segmentation performance, has long been recognized as a difficult task. For a segmentation validation task, it may be effective to compare the results of an automatic segmentation algorithm to multiple expert segmentations. Recently an expectation-maximization (EM) algorithm for simultaneous truth and performance level estimation (STAPLE) was developed to this end to compute both an estimate of the reference standard segmentation and performance parameters from a set of segmentations of an image. The performance is characterized by the rate of detection of each segmentation label by each expert in comparison to the estimated reference standard. This previous work provides estimates of performance parameters,but does not provide any information regarding the uncertainty of the estimated values. An estimate of this inferential uncertainty, if available, would allow the estimation of confidence intervals for the values of the parameters. This would facilitate the interpretation of the performance of segmentation generators and help determine if sufficient data size and number of segmentations have been obtained to precisely characterize the performance parameters. We present a new algorithm to estimate the inferential uncertainty of the performance parameters for binary and multi-category segmentations. It is derived for the special case of the STAPLE algorithm based on established theory for general purpose covariance matrix estimation for EM algorithms. The bounds on the performance parameters are estimated by the computation of the observed information matrix.We use this algorithm to study the bounds on performance parameters estimates from simulated images with specified performance parameters, and from interactive segmentations of neonatal brain MRIs. We demonstrate that confidence intervals for expert segmentation performance parameters can be estimated with our algorithm. We investigate the influence of the number of experts and of the segmented data size on these bounds, showing that it is possible to determine the number of image segmentations and the size of images necessary to achieve a chosen level of accuracy in segmentation performance assessment.

Evaluation of uterine cervix segmentations using ground truth from multiple experts

This work is focused on the generation and utilization of a reliable ground truth (GT) segmentation for a large medical repository of digital cervicographic images (cervigrams) collected by the National Cancer Institute (NCI). NCI invited twenty experts to manually segment a set of 939 cervigrams into regions of medical and anatomical interest. Based on this unique data, the objectives of the current work are to: (1) Automatically generate a multi-expert GT segmentation map; (2) Use the GT map to automatically assess the complexity of a given segmentation task; (3) Use the GT map to evaluate the performance of an automated segmentation algorithm. The multi-expert GT map is generated via the STAPLE (Simultaneous Truth and Performance Level Estimation) algorithm, which is a well-known method to generate a GT segmentation from multiple observations. A new measure of segmentation complexity, which relies on the inter-observer variability within the GT map, is defined. This measure is used to identify images that were found difficult to segment by the experts and to compare the complexity of different segmentation tasks. An accuracy measure, which evaluates the performance of automated segmentation algorithms is presented. Two algorithms for cervix boundary detection are compared using the proposed accuracy measure. The measure is shown to reflect the actual segmentation quality achieved by the algorithms. The methods and conclusions presented in this work are general and can be applied to different images and segmentation tasks. Here they are applied to the cervigram database including a thorough analysis of the available data.