The preliminary stage in developing an artificial intelligence algorithm: a study of the inter- and intra-individual variability of phase annotations in internal fixation of distal radius fracture videos

Abstract

Aim: As a preliminary stage in the development of an artificial intelligence (AI) algorithm for surgery, this work aimed to study the inter- and intra-individual variability of phase annotations in videos of minimally invasive plate osteosynthesis of distal radius fractures (MIPO). The main hypothesis was that the inter-individual variability was almost perfect if Cohen's kappa coefficient (k) was ≥ 81% overall; the secondary hypothesis was that the intra-individual variability was almost perfect if the F1-score (F1) was ≥ 81%. Methods: The material comprised 9 annotators and three annotated MIPO videos with 5 phases and 4 sub-phases. Each video was presented 3 times to each annotator. The method involved analysing the inter-individual variability of annotations by computing k and F1 from a reference annotator. The intra-individual variability of annotations was analysed by computing F1. Results: Annotation anomalies were noticed: either absences or differences in phase and sub-phase annotations. Regarding the inter-individual variability, an almost perfect agreement between annotators was observed because k ≥ 81% for the three videos. Regarding the intra-individual variability, F1 ≥ 81% for most phases and sub-phases with the nine annotators. Conclusion: The homogeneity of annotations must be as high as possible to develop an AI algorithm in surgery. Therefore, it is necessary to identify the least efficient annotators (measurement of the intra-individual variability) to provide them with individual training and a personalised annotation rhythm. It is also important to optimise the definition of the phases, improve the annotation protocol and choose suitable training videos.