Iterative Refinement to Improve Data Quality and Label Consistency by Synergizing Parsimony Model Guidance and Physician Interaction

Abstract

<h3>Purpose/Objective(s)</h3> Clinical data and labeling are usually noisy and uncertain. Modeling based on such inconsistent data risks overfit and may generate faulty insight on prominent features and regression relationships. We hypothesize inter-sample consistency serves as a rational surrogate for data quality and can be used to guide improvement, in the absence of an absolute ground-truth. To this end, we aim to synergize a deep-learning setting and an iterative interactive refinement procedure for label refinement. <h3>Materials/Methods</h3> We proposed a novel approach by guiding physician-approved label perturbation with parsimonious deep network modeling. The proposed procedure alternates between fitting a parsimonious deep network model to human labels and guiding the human observer to review the labels to identify ill-fitted examples to perform label refinement when appropriate. Convergence is claimed when further iteration stops to produce clinically pronounced improvement, as defined from a statistical equivalence test. To evaluate the efficacy of the refinement, input-output agreement index (IOAI) based on partial ranking consistency is also calculated. We took as a use case in segmenting the lumen and vessel wall from MR vessel wall imaging (VWI) from 80 patients with intracranial atherosclerotic disease, on four locations with high likelihood of plaque presence: the intracranial internal carotid artery, the middle cerebral artery, the intracranial vertebral artery, and the basilar artery. Each segment contained 30 contiguous 2D cross-sectional slices with 0.55 mm slice thickness and 0.10 mm in-plane resolution. A lightweight 2.5D segmentation network was used as the low-dimensional model, and equivalence criterion was defined by one-sided superiority threshold of 0.03 in Dice similarity coefficient (DSC) based on reported performance from existing study. Clinical soundness was further assessed with the variation in normalized wall index as a quantitative imaging index, expected to low if segmented structures were piecewise smooth conforming to clinical insight. <h3>Results</h3> 5-fold cross validation based on the 80*4*30 slice samples showed enhanced modeling performance and better conformality to clinical insight with the final labels. In this use case, DSC improved from 0.893±0.108 to 0.938±0.078 for lumen, from 0.806±0.086 to 0.879±0.072 for vessel wall, the total variation in normalized wall index decreased from 0.757±0.181 to 0.586±0.182, and the input-output agreement index increased from 0.523 to 0.556 by the proposed refinement procedure. <h3>Conclusion</h3> This study demonstrates that inconsistency in clinical labels and physiological limitations can be addressed with an iterative interactive process with parsimonious modeling, taking advantage of flexibility from deep networks. The rationale generalizes to other label settings and tasks.