COVID-19 restrictions and the instructions to stay at home (SaH) may have had an impact on child behavior including physical activity (PA) and health-related quality of life (HRQoL) in Switzerland. Therefore, we investigated PA and HRQoL during and after the SaH in 57 Swiss school children aged 7 to 12 years (M = 10.44; SD = 1.34). PA was measured using accelerometry (Actigraph GT3X) and HRQoL using the Kid-KINDLR questionnaire. During and post data was compared using paired sample t-tests. Independent t-tests were used to compare the HRQoL of physically active children with non-physically active children. PA in light (d = −0.56), moderate (d = −0.44), moderate-to-vigorous (d = −0.28) as well as overall HRQoL (d = −0.66), psychological well-being (d = −0.48), self-esteem (d = −0.39), friends (d = −0.70) and everyday functioning (d = −0.44), were significantly lower during SaH than afterwards. Children who adhered to PA recommendations (+60 min of moderate-to-vigorous PA) during SaH had a better overall HRQoL (d = 0.61) and psychological well-being (d = 0.56) than those who did not. Since PA levels and HRQoL were lower during SaH compared to afterwards, it seems that the restrictions negatively impacted children’s PA and HRQoL. During future SaHs, promoting children’s PA and HRQoL seems important.