O-242 Comparing the performance of an artificial intelligence model for predicting embryo implantation between clinics with patient cohorts of different maternal age distributions

Abstract

Abstract Study question Do differences in maternal age distributions contribute to the variability in the discrimination performance of an embryo viability model based on artificial intelligence (AI)? Summary answer Using a common reference age distribution, enables a less biased and thus more standardized comparison of embryo implantation prediction between clinics. What is known already Studies from different clinics have shown a quite large variation in the performance of the same AI model. The most common method of comparing performance between clinics is to use simple clinic-specific performance statistics while ignoring dissimilarities between clinics in patient cohorts, culture conditions etc. A more appropriate method is meta-analysis of performance across clinics rather than using overall performance, however, this does also not account for dissimilarities between clinics. Assessing the performance in subgroups, although common practice, reduces the sample size significantly and limits interpretation to each separate subgroup. Study design, size, duration This is a retrospective observational multi-site study with data from four clinics collected over varying time periods. In all participating clinics embryos were cultured in EmbryoScope time-lapse incubators (ES-D, ES+, ES-Flex; Vitrolife). Embryo implantation likelihood was evaluated with the iDAScore v1.0 AI decision support tool. Data from 4,805 single embryo transfers on day 5 and 6 (fresh and frozen) with known fetal heartbeat outcome were included. Donor egg cycles were strictly excluded. Participants/materials, setting, methods Performance of embryo implantation prediction was measured in terms of areas under the ROC curve (AUC). To account for age differences, we performed a standardization of the AUCs using a weighted version of the ROC curves with weights defined as the ratio between the age densities in a reference population and in the specific clinic. The reference population was comprised of data that was held out from the original model development. Main results and the role of chance Different age distributions with medians ranging from 32 to 40 years were observed in the four clinics. The unweighted overall AUC was 0.65 (95% CI: 0.64 to 0.69), whereas the AUC in the reference population was 0.67 (95% CI: 0.63 to 0.71). The predictive performance of the iDAScore model varied between clinics with the unweighted AUCs ranging from 0.58 to 0.69. Meta-analysis across all four clinics showed a summary average of 0.65 (95% CI: 0.59 to 0.70). We then standardized to the age distribution in a reference population in which the maternal age ranged from 21 to 44 years with a median of 38. The values of the AUCs after standardization ranged from 0.60 to 0.71 between clinics with a summary average of 0.67 (95% CI: 0.62 to 0.72). The standard error of the summary average was 5% lower when standardization was used and the estimated variation between clinics in the meta-analysis was 16% lower. In conclusion, we have shown with our standardized analysis method that the variability in embryo performance prediction between clinics can be partially explained by the difference in maternal age distributions. Limitations, reasons for caution The summary average of the AUC is highly dependent on the age distribution in the chosen reference population. Therefore, the exact value of this estimate should be interpreted with caution. However, comparing AUCs between clinics using this standardization approach is less biased by differences in age distribution. Wider implications of the findings Simple comparisons of the performance of AI-based embryo implantation prediction in different clinics may be influenced by factors such as age. Using a standardization approach, this influence can be mitigated, enabling a less biased comparison. Differences in other important factors apart from age can potentially also be considered. Trial registration number not applicable