Background: This research aimed to investigate the correlation between students’ physical fitness and campus-built environment, which could put forward some suggestions for the construction of a campus environment. Method: Four colleges in Xi’an were regarded as special “semi-closed” spaces. Combined with ArcGIS and SPSS, the correlation between the built environment of colleges and the students’ physical fitness test results in 2019 was analyzed (n = 1498). Results: regarding the men questioned in this research, there was a significant correlation between street connectivity and vital capacity, grip strength, 50 m running, 1000 m running, a significant correlation between land use mix and vital capacity, sit-and-reach, pull-up, grip strength, a significant correlation between green space per capita and vital capacity, grip strength, 50 m running, and a significant correlation between walk score and vital capacity, pull-up, grip strength, and 50 m running. Regarding the women questioned in this research, there was a significant correlation between street connectivity and vital capacity, grip strength, 50 m running, 800 m running, curl-up, a significant correlation between land use mix and vital capacity, sit-and-reach, curl-up, grip strength, 800 m running, a significant correlation between green space per capita and vital capacity, grip strength, curl-up, sit-and-reach, and a significant correlation between walk score and vital capacity, curl-up, grip strength, and 800 m running. Conclusion: the built environment on campus can indirectly affect the physical fitness of college students. Increasing the number of intersections and short connections of campus streets, ensuring that the green space of the campus meets the standards, and reasonably arranging the site selection of buildings are conducive to improving the physical fitness of students.