BackgroundCurrent guidelines for mammography screening for breast cancer vary across agencies, especially for women aged 40–49. Using artificial Intelligence (AI) to read mammography images has been shown to predict breast cancer risk with higher accuracy than alternative approaches including polygenic risk scores (PRS), raising the question whether AI-based screening is more cost-effective than screening based on PRS or existing guidelines. This study provides the first evidence to shed light on this important question.MethodsThis study is a model-based economic evaluation. We used a hybrid decision tree/microsimulation model to compare the cost-effectiveness of eight strategies of mammography screening for women aged 40–49 (screening beyond age 50 follows existing guidelines). Six of these strategies were defined by combinations of risk prediction approaches (AI, PRS or family history) and screening frequency for low-risk women (no screening or biennial screening). The other two strategies involved annual screening for all women and no screening, respectively. Data used to populate the model were sourced from the published literature.ResultsRisk prediction using AI followed by no screening for low-risk women is the most cost-effective strategy. It dominates (i.e., costs more and generates fewer quality adjusted life years (QALYs)) strategies for risk prediction using PRS followed by no screening or biennial screening for low-risk women, risk prediction using AI or family history followed by biennial screening for low-risk women, and annual screening for all women. It also extendedly dominates (i.e., achieves higher QALYs at a lower incremental cost per QALY) the strategy for risk prediction using family history followed by no screening for low-risk women. Meanwhile, it is cost-effective versus no screening, with an incremental cost-effectiveness ratio of $23,755 per QALY gained.ConclusionsRisk prediction using AI followed by no breast cancer screening for low-risk women is the most cost-effective strategy. This finding can be explained by AI’s ability to identify high-risk women more accurately than PRS and family history (which reduces the possibility of delayed breast cancer diagnosis) and fewer false-positive diagnoses from not screening low-risk women.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12885-022-09613-1.