Abstract The identification and quantification of prey ingested is a limiting factor in trophic ecology studies and is fundamental for assessing the impact of a predator on prey populations. Vendace (Coregonus albula) and whitefish (C. lavaretus) are two congeneric species, which are commonly preyed on by Baltic ringed seals (Pusa hispida). The otoliths of these two species are, however, very similar and distinguishing between them in the seal diet using visual inspection has so far been challenging. Here, otolith shape outline analyses were used in combination with machine learning techniques to discriminate between eroded vendace and whitefish otoliths from ringed seal diet samples. An experiment of in vitro digestion of the otoliths was performed to train a machine learning model. Our model is able to self-assign known digested otoliths back to their species of origin with >90% accuracy. Furthermore, 89% (N = 690) of the otoliths collected from digestive tract samples could be successfully assigned to species level, i.e. vendace or whitefish. This method is readily applicable for improved understanding of ringed seal feeding habits and predator–prey interactions, as well as large-scale applications to generate seal-predation matrix inputs for stock assessments of vendace and whitefish. Further development of the machine learning techniques to discriminate between prey species in seal and other piscivorous diets is strongly encouraged.