Background: Data on the differences in functional recovery between inpatient rehabilitation for total hip arthroplasty (THA) and total knee arthroplasty (TKA) are lacking, and the factors influencing the improvement of short-term functional mobility remain unknown. In this study, we compared the short-term functional outcomes of both procedures and identified early postoperative predictors of physical function gain during post-acute rehabilitation. Methods: A total of 435 patients who underwent THA and TKA were included. The main outcomes were knee extension strength, the motor component of the Functional Independence Measure, Numerical Rating Scale, 10 Meter Walk Test, Timed Up and Go (TUG) test, and the Berg Balance Scale. The recovery process and rehabilitation outcomes were compared between patients with THA and TKA. Additionally, predictors related to physical performance improvement were examined for each procedure. Results: Patients with THA and TKA achieved significant short-term functional recovery after multidisciplinary rehabilitation (time; p < 0.001). However, the pain score was higher at discharge in patients with TKA (p < 0.001). Age (β: −0.264, p = 0.009) and TUG test (β: −0.884, p < 0.001) in THA, and non-operated knee extension strength (β: 0.234, p = 0.016) and TUG test (β: −0.783, p < 0.001) in TKA were significant early postoperative predictors of functional mobility. Conclusions: Multidisciplinary rehabilitation was beneficial for functional improvement in patients with THA and TKA despite persistent pain at discharge after TKA. Baseline functional levels in both groups and non-operated knee extension strength in TKA can be useful performance-based predictors of short-term gait function improvement.