Hip fracture patients often suffer from pain for several months after surgery. This may lead to physical inactivity and subsequent mobility limitation and disability. The purpose of this study was to investigate the association between severe musculoskeletal pain and the level of physical activity in older people with a history of hip fracture. Data were collected from 60- to 85-year-old (n = 78) community-dwelling people, 0.7–7.5 years after hip fracture. Physical activity was assessed with the Yale Physical Activity Survey (YPAS). According to the YPAS summary index distribution, the participants were divided into the physically inactive and physically active groups. Musculoskeletal pain in the lower back, hip or knee region was assessed by the Visual Analog Scale. Pain was defined as severe pain if the subject rated the pain 66 mm or over (upper third of the scale) in at least one of the relevant body regions in the lower body. Thirty-three subjects (42%) experienced severe pain in the lower body, of whom 23 (70%) reported severe pain in the operated hip. Twenty-three patients (30%) used pain medication prescribed by a physician. The level of musculoskeletal pain was significantly higher in the physically inactive (62%) than in the physically active (31%) group (p = 0.011). Logistic regression analysis showed that people with severe pain had over three times (OR: 3.5, 95% CI: 1.30–9.39) the risk for physical inactivity compared to those with less or no pain. Multivariate adjustments for balance confidence, time since fracture, number of chronic diseases and type of surgery did not materially change the estimate (OR: 3.0, 95% CI: 1.00–9.30). Pain is an important associated factor of physical inactivity in older people with a hip fracture history. Pain management may be important in restoring and sustaining the level of physical activity after hip fracture. Further prospective and experimental studies are, however, needed to explore the causality between pain and physical activity as well as the role of pain management and physical activity interventions in preventing mobility limitation and disability among older people recovering from hip fracture.