Decreased physical function with increasing life expectancy is a public health concern worldwide. Knee osteoarthritis (KOA) is considered one of the primary illnesses causing decreased physical function. Depression affects decreased physical function and is closely related to knee pain in KOA. However, the effect of these interacting factors on physical function is not clear. We conducted a cross-sectional analysis of the baseline data of 1106 subjects of the 2009 Locomotive Syndrome and Health Outcome in Aizu Cohort Study (LOHAS). We determined the association between their Timed Up and Go test (TUG) scores and radiographic KOA, knee pain, and depression in a multivariate analysis. Severe knee pain was significantly associated with decreased physical function (the odds ratio [OR] was 2.13, 95% confidence interval [CI]: 1.32-4.89), as was depression (OR 2.64, 95% CI 1.61-4.33). Only Kellgren-Lawrence (KL) grade 4 was significantly associated with decreased physical function in the radiographic KOA severity (OR 6.58, 95% CI 1.75-24.68). Severe knee pain and depression were significantly associated with decreased physical function, but not radiographic KOA severity except for KL grade 4. The limitations of using radiographic KOA severity alone as the indicator of assessment for physical function were suggested. When assessing decreased physical function, the clinical focus tends to be on radiographic KOA severity, but it is important to consider the patient's knee pain and psychological factors.