To address behavioral interferences such as head turning and lowering during rumination in group-housed dairy cows, an enhanced network algorithm combining the YOLOv5s and DeepSort algorithms was developed. Initially, improvements were made to the YOLOv5s algorithm by incorporating the C3_CA module into the backbone to enhance the feature interaction and representation at different levels. The Slim_Neck paradigm was employed to strengthen the feature extraction and fusion, and the CIoU loss function was replaced with the WIoU loss function to improve the model’s robustness and generalization, establishing it as a detector of the upper and lower jaws of dairy cows. Subsequently, the DeepSort tracking algorithm was utilized to track the upper and lower jaws and plot their movement trajectories. By calculating the difference between the centroid coordinates of the tracking boxes for the upper and lower jaws during rumination, the rumination curve was obtained. Finally, the number of rumination chews and the false detection rate were calculated. The system successfully monitored the frequency of the cows’ chewing actions during rumination. The experimental results indicate that the enhanced network model achieved a mean average precision (mAP@0.5) of 97.5% and 97.9% for the upper and lower jaws, respectively, with precision (P) of 95.4% and 97.4% and recall (R) of 97.6% and 98.4%, respectively. Two methods for determining chewing were proposed, which showed false detection rates of 8.34% and 3.08% after the experimental validation. The research findings validate the feasibility of the jaw movement tracking method, providing a reference for the real-time monitoring of the rumination behavior of dairy cows in group housing environments.