Time-resolved vortices are indispensable rotational flow structures which impact fuel-air mixing, combustion, and thermal efficiency of internal combustion engine. In-cylinder flow fields are largely stochastic, therefore capturing full spatial and temporal details of in-cylinder vortex behavior not only requires enormous datasets, but also time-efficient analysis methods. The loss of transient vortex information often becomes problematic due to temporal averaging of spatial vortex features. In this study, a two-step vortex pattern detection and quantification scheme with a novel application of K-means algorithm was implemented to investigate the transient vortex dynamics and cyclic variations inside an optical direct injection engine. A vortex pattern recognition method was first implemented to locate the vortices which appeared intermittently in each cycle. Then, the K-means algorithm was applied to cluster the vortex zone characteristics of consecutive engine cycles. Results show that vortex detection based on K-means algorithm can efficiently and accurately cluster the vortex centers into different zones and quantify their cyclic variations. It is also capable of minimizing the issue of vortex misidentification due to temporal averaging. This work demonstrates the application of K-means algorithm to detect the dynamic nature of vortex motions and provides a quantitative analysis on flow vortex characteristics and cyclic variations.