Background/Objectives: This cross-sectional study aimed to investigate the relationship between swallowing function and low serum albumin levels in older Japanese people (aged ≥ 75 years). Methods: A total of 3258 participants (1325 males, 1933 females; mean age, 80.8 years) who had a dental checkup in Gifu City, Japan, between April 2020 and March 2021 were recruited. Swallowing function was assessed using the repetitive saliva swallow test, with poor swallowing function defined as swallowing fewer than three times in 30 s. A serum albumin level < 3.6 G/dL was considered low. Results: In total, 78 participants (2%) had a low serum albumin level. Furthermore, after adjusting for age, gender, circulatory disease, support/care-need certification, number of present teeth, and tongue and lip function, a low serum albumin level was positively associated with older (odds ratio [OR]: 1.115; 95% confidence interval [CI]: 1.064–1.169), male (OR: 2.208; 95% CI: 1.360–3.584), circulatory disease (OR: 1.829; 95% CI: 1.079–3.099), support/care-need certification (OR: 2.087; 95% CI: 1.208–3.606), and poor swallowing function (OR: 2.379; CI: 1.377–4.112). Conclusions: These results indicate that poor swallowing function was associated with a low serum albumin level in older Japanese people aged ≥ 75 years.