The Physical Exercise Rating Scale, Mindfulness Attention Awareness Scale (MAAS), Ruminative Thinking Scale (RRS), and Pittsburgh Sleep Quality Index (PSQI) are used to conduct the questionnaire among a sample of 1,006 college students (average age = 19.95 years, SD = 1.86 years) to figure out whether there is any correlation between physical exercise and sleep quality in college students as well as how the mechanism of mindfulness and ruminative thinking plays a role in them. For data analysis, the Harman single-factor test was used; for the common method deviation test, Pearson’s correlation analysis, and the mediating effect tested by using the bootstrap method were carried out. Results: (1) Overall, 34% (PSQI ≥ 8) of college students’ sleep quality is poor. (2) The PSQI is positively correlated with ruminative thinking, while both are negatively correlated with the level of physical exercise and mindfulness; the level of physical exercise is positively correlated with the level of mindfulness. (3) Physical exercise can significantly negatively predict sleep quality (β = −0.08, P < 0.05), significantly positively predict mindfulness (β = 0.12, P < 0.001), and significantly negatively predict ruminative thinking (β = −0.07, P < 0.05). When participate in physical exercise, mindfulness, and ruminative thinking enter the regression equation at the same time, only mindfulness and ruminative thinking can predict sleep quality(β = −0.15, P < 0.001) significantly negatively predicted sleep quality and ruminative thinking (β = 0.22, P < 0.001) significantly positively predicted sleep quality, while physical exercise (β = −0.04, P > 0.05) had no significant predictive effect on sleep quality. (4) After controlling for age and gender, physical exercise appears to have an impact on sleep quality of college students through the independent intermediary role of mindfulness and ruminative thinking and the chain intermediary role of mindfulness and ruminative thinking, with a total mediating effect value of −0.007. This study reveals the relationship between physical exercise and sleep quality, as well as its mechanism, thus guiding college students to actively participate in physical exercise. It also provides corresponding suggestions to improve sleep quality as well as physical and mental health in college students.