Annotator bias and its effect on deep learning segmentation of uncured composite micrographs

Abstract

This study demonstrated substantial agreement between groups of experts and non-experts in labelling image regions required for semantic segmentation of X-ray micrographs of uncured composite prepregs. Twelve participants, six experts and six non-experts, were given a 1-h training session covering the three different image regions that are present in an uncured composite micrograph: voids, dry fibre areas, and filled fibre areas. High consensus was observed in the centre of objects, but disagreement in labelling between the groups was observed at the interphase regions where the grey level intensity becomes ambiguous in these low-contrast images, such as at the edges of the dry fibre areas and voids. Also, labelling small interlaminar voids caused disagreement. The participants highlighted the role of software by reporting a preference to defining the vertices of a polygon over colouring-in regions of image segments. The resulting Deep Learning segmentation with expert and non-expert group labels were in almost perfect agreement, as measured by the Fleiss Kappa coefficient, and was able to segment voids and dry fibre areas better than thresholding.