The COVID-19 pandemic and the imposed social distancing measures caused negative changes in physical activity levels (PALs) and physical fitness (PF) among adolescents, but the potential mediating effect of the pandemic on the association between PAL and PF is unknown. This study aimed to evaluate gender-specific associations between objectively measured PAL and indices of PF among high school adolescents during the period of the COVID-19 pandemic. The participants were 150 adolescents (101 females) aged 14-18 years; their anthropometrics were evaluated, and they were tested on PF (cardiorespiratory fitness (CRF; beep test), power (broad jump), flexibility (sit-and-reach test), and abdominal strength (sit-ups)) and PAL (using a pedometer for 7 days) during the pandemic period. A T-test was calculated to determine differences between genders. Associations between variables were evaluated using Pearson correlations. Additionally, multivariate taxonomic classification was used to sort participants into homogenous groups (clusters) according to their PF, and then analysis of variance (ANOVA) was performed to differentiate them according to PAL. For the total sample, PAL was correlated with CRF only (R = 0.25, p < 0.05), while gender-stratified correlations showed that PAL was significantly associated with CRF among girls only (R = 0.29, p < 0.05), which was additionally confirmed with multivariate cluster analysis and subsequent ANOVA. No association between PAL and PF was found for boys. The relatively low association between PAL and PF is most likely related to the mediating effect of the change in life circumstances during the COVID-19 pandemic, and limited applicability of pedometers in evaluating high-intensity PAL. Further studies in other age groups and environments are warranted.