Abstract Vortex core detection remains an unsolved problem in the field of experimental and computational fluid dynamics. Available methods such as the Q, delta, and swirling strength criterion are based on a decomposed velocity gradient tensor but detect spurious vortices (false positives and false negatives), making these methods less robust. To overcome this, we propose a new hybrid machine learning approach in which we use a convolutional neural network to detect vortex regions within surface streamline plots and an additional deep neural network to detect vortex cores within identified vortex regions. Furthermore, we propose an automatic labeling approach based on K-means clustering to preprocess our input images. We show results for two classical test cases in fluid mechanics: the Taylor–Green vortex problem and two rotating blades. We show that our hybrid approach is up to 2.6 times faster than a pure deep neural network-based approach and furthermore show that our automatic K-means clustering labeling approach achieves within 0.45% mean square error of the more labour-intensive, manual labeling approach. At the same time, by using a sufficient number of samples, we show that we are able to reduce false positives and negatives entirely and thus show that our hybrid machine learning approach is a viable alternative to currently used vortex detection tools in fluid mechanics applications.