Abstract Background Children begin using digital sources of health information at a young age and are quickly exposed to a variety of unmoderated health content. Health literacy helps children navigate, understand, evaluate, and act on this content. But is the educational system preparing young people for these tasks? This study develops a validated, performance-based measure of health literacy and applies it to a sample of 9- to 13-year-old children (grades 4 to 7). Methods A performance-based tool to measure children’s health literacy was developed through expert consultation and qualitative and quantitative pretesting. The tool uses vignettes and quizzes and captures three dimensions of health literacy: nutrition, physical activity, and psychosocial health. The validated tool was then used in a cross-sectional survey of Austrian children aged 9 to 13 years (n = 780): Diet, Physical Activity, and Psychosocial Health. Results Results show that the performance-based health literacy scores do not correlate with self-reported scores. Girls (vs. boys) and native speakers perform better on the performance-based literacy test, while boys and children reporting more robust financial backgrounds overestimate their literacy on self-reported questions. Most importantly, the data do not show meaningful increases in performance-based health literacy across grade levels. For nutrition literacy, the study even shows a slight negative trend across grade levels. Conclusions This study questions the heavy reliance on self-reported health literacy, which is often confounded with self-efficacy and social environment. Performance-based instruments are needed to measure health literacy as an educational outcome. This study presents one of the first validated instruments. Most importantly, the study shows that the current educational system may not produce meaningful increases in performance-based health literacy, calling for deeper educational reforms. Key messages • Performance-based health literacy is a distinct concept from the subjective health literacy measures that dominate current health literacy research. • Performance-based health literacy did not increase with grade level, indicating a poor performance of the education system in strengthening health literacy.