Health literacy has been reported to have effects on health behavior change and health-related outcomes, but few studies have explored the association between health literacy and frailty. The aim of our study is to investigate the relationships between health literacy and frailty among community-dwelling seniors. This cross-sectional study enrolled 603 community-dwelling older adults (307 women) in residential areas, with a mean age of 70.9 ± 5.82 years. Health literacy was assessed using the Mandarin version of the European Health Literacy Survey Questionnaire. Physical frailty was defined by Fried frailty phenotype. Logistic regression was carried out to determine potential risk factors of frailty. In the multivariate logistic regression model, physical activity (Odds Ratio [OR] 1.47, 95% Confidence Interval [CI] 1.06–2.03) and health literacy (sufficient vs. excellent: OR 2.51, 95% CI 1.32–4.77) were associated with prefrailty and frailty. In subgroup analysis, pre-frailty and frailty were also negatively associated with health literacy in individuals with ‘insufficiently active’ (inadequate vs. excellent: OR 5.44, 95% CI 1.6–18.45) and ‘sufficiently/highly active’ physical activity levels (sufficient vs. excellent: OR 2.41, 95% CI 1.07–5.42). Therefore, in these community-dwelling elderly adults, health literacy was associated with pre-frailty and frailty regardless of age, gender, socio-economic status, and education level.