Background: This study sought to investigate whether different physical function tests (objective measures of physical performance) may identify a low physical resilience in breast cancer survivors (BCS). Methods: This analytical cross-sectional study evaluated 146 BCS and 69 age-matched women without breast cancer history. The different times after the end of treatment were used as criteria for group division. Participants were divided into four groups: control (CT: n = 69–women without breast cancer history); <1.0 years after the end of treatment (<1 YAT: n = 60); 1–3.9 years after the end of treatment (1–3.9 YAT: n = 45); and ≥4 years after the end of treatment (>4 YAT: n = 41). Physical function was evaluated by 4 m walk test (4-MWT), five-times-sit-to-stand test (FTSST), timed up and go test (TUG), and short physical performance battery (SPPB). Age, menopausal status, smoking, number of medications, level of physical activity, body mass index, and muscle strength were used as confounding variables in ANCOVA. Results: All groups that underwent cancer treatment (<1 YAT, 1–3.9 YAT and ≥4 YAT) had lower physical performance (p < 0.001) identified by 4 MWT, TUG, and FTSST when compared to the CT group. For the SPPB, the <1 YAT and ≥4 YAT groups had lower performance (p = 0.005) when compared to the CT. Conclusions: The different physical function tests can be used to identify a low physical resilience in BCS.