P-269 A validation study for artificial intelligence (AI) compared with manual annotation, using donor eggs reveals that AI accurately predicts blastulation

Abstract

Abstract Study question Are the annotations produced by AI comparable to manual annotations? Does AI accurately assess fertilisation checks, and predict embryo usage and blastulation compared to embryologists? Summary answer Automatic annotations by AI was consistent with manual annotations. AI implantation algorithms had strong prediction of blastulation and embryo usage. What is known already Currently, embryos are manually annotated for specific morphokinetic features during embryo development. This is a labour-intensive process, and dependent on training and experience, leading to inter and intra clinic variation. The decision to transfer, freeze or discard embryos relies heavily on these annotations. It is paramount that we develop a tool that will provide consistency and accuracy in annotation and produce scores that can facilitate decisions around embryo usage. AI has demonstrated its potential to achieve this, but first must be validated before its integration into clinical practice. There have been no such studies demonstrating this so far. Study design, size, duration Retrospective cohort study, that took place between September to December 2021 at a private fertility clinic in Spain. To control for embryo variability, this study only included 179 time-lapse videos for embryos created from donor eggs. This was based on the understanding that donor eggs are more likely to produce better quality blastocysts and embryos and thus will give the most optimal conditions for annotation in a validation framework. Participants/materials, setting, methods The same time-lapse cultured embryos were annotated manually and automatically by CHLOE(Fairtility, an AI-based tool). Manual and CHLOE annotations were compared to assess the strength of agreement (i) using intra-class correlation (ICC), and (ii) the proportion of corrections required at the pronuclei (PN) stage. AI accuracy in predicting blastulation at 30hours, and blastulation before 116 hours, was also assessed using AUC as the efficacy metric. Embryo usage was compared with the AI-generated ranking of embryos. Main results and the role of chance The majority of morphokinetic variables showed a very-strong agreement, with an ICC range of (0.81-1.00), namely for; tPNf, t2, t3, t5, t7, tSB, tB and tEB. Only t4 (0.5) showed a moderate agreement. On average (Mean+-Standard deviation), AI annotated t4 later than embryologists (36+-5vs39+-10 (hours)). All other variables fell within a strong ICC of (0.61-0.8). There were no very weak (0-0.2) or weak (0.21-0.4) variables. PN agreement between AI and embryologists was 93%: PN’s had to be corrected by an embryologist only 7%(n = 179) of the time. AI predicted blastulation on day 3 with a high level of sensitivity 0.77 and specificity 0.82, (AUC: 0.84,p<0.0001). Furthermore, the blastulation score given on day 3 was a predictor of blastulation before 116 hours with a high sensitivity 0.77 and specificity 0.80, (AUC: 0.81,p<0.0001). Similarly, AI-generated ranking accurately correlated with embryologist decisions to freeze, transfer or discard embryos, with an overall high sensitivity 0.88 and specificity 0.67, (AUC: 0.84,p<0.0001). A rank of 1 was seen in 14%(n = 113) of embryos, all of which were frozen or transferred. Some embryos that scored a rank of 2 were discarded, but this was significantly lower than those that scored a rank of 3 or more (3%vs32%,p=0.0004). Limitations, reasons for caution This study only included embryos from donor eggs. Furthermore, this study occurred at a single site and is planned to be replicated at several clinics. Where there are discrepancies between human and AI, further studies are required to determine the ground truth. Wider implications of the findings This study demonstrates an AI framework to safely introduce AI in the fertility clinic. AI will accurately annotate embryos and give reliable scores to predict good quality blastulation, and inform decisions around embryo usage determination. AI provides a time-effective, objective tool in decision-making, with the potential to optimise success. Trial registration number not applicable