Recess time (RT) is a main component of school-based activities, and could contribute up to 40% of the physical activity (PA) recommended in the health guidelines. The main goal was to analyze the association between accelerometer-measured PA and sedentary time during RT with cardiorespiratory fitness (CRF). A total of 146 children and adolescents, aged 8–19 years, were recruited from six schools. PA levels were measured with GT3X accelerometers over 7 days. CRF was measured using a portable breath-by-breath gas analyzer. A general linear model (GML) was conducted to analyze the association between PA intensities and CRF during RT. Additionally, a logistic binary regression was used to evaluate the risk of unhealthy CRF among different categories of PA and sedentary time. Participants classified as healthy showed higher PA levels during RT than those classed as unhealthy. GML analysis showed that sedentary time during RT was significantly associated with VO2Peak. Finally, compared to individuals accumulate less than 15-min of ST during recess, individuals who were sedentary for more than 15 min during RT presented 43.78 times of having unhealthy CRF (95% CI 3.873–494.824). Our data suggest an association between recess sedentary time and unhealthy CRF. Thus, school-aged children and adolescents must be empowered to perform PA during RT to prevent the deleterious effects of sedentary time on CRF.