Physical fitness is one of the most important health indicators in children. Although appropriate body composition or certain lifestyle factors such as frequent physical activity are thought to improve physical fitness, results of previous studies are inconsistent, and most studies were from Western countries. We investigated associations of body composition and modifiable lifestyle factors such as physical activity, screen time, diet and sleep duration with physical fitness in Japanese primary school children. 2308 children (age 10-12 years old) in 12 primary schools were analysed in this cross-sectional study. Physical fitness was evaluated by sports battery tests conducted routinely and annually at schools. The total score of sports battery tests, 20-m shuttle run (laps) and grip strength (kg) were selected as outcomes. Information about lifestyle factors was collected by two questionnaires. Associations between lifestyle factors and physical fitness were assessed by multivariable linear mixed models by sex. Frequent exercise was related to better overall physical fitness. Regarding the 20-m shuttle run, many unfavourable lifestyle factors such as higher BMI in boys (β -7.37, 95% confidence interval [CI] -8.39, -6.35) and girls (β -3.54, 95% CI -4.50,-2.58), longer screen time (β -4.31, 95% CI -7.29, -1.34) in boys and girls (β -5.65, 95% CI -9.01, -2.30); shortest (reference) versus longest, breakfast skipping in boys (β -5.24, 95% CI -8.71, -1.77) and girls (β -3.57, 95% CI -6.84, -0.30); consumers (reference) versus skippers were associated with worse performance. Better quality of diet was associated with better results in the 20-m shuttle run only in girls (β 2.58, 95% CI 0.24, 4.93); lowest (reference) versus highest. Frequent exercise was related to better physical fitness. Higher BMI and unfavourable lifestyle factors such as longer screen time and breakfast skipping were associated with worse results of the 20-m shuttle run.