Introduction The aims of this study were to identify homogeneous subgroups with distinct trajectories of pain, physical function and physical activity (PA) components in patients with symptomatic knee and/or hip osteoarthritis (OA) and to identify the baseline predictive factors associated with these trajectories. Methods The KHOALA cohort is a French population-based multicenter cohort of 878 patients with symptomatic knee and/or hip OA, aged between 40 and 75 years old recruited between 2007 and 2009. Six outcomes assessed annually during a period of 5 years were modeled in the multi-trajectory model. Pain and physical function were measured with the Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) questionnaire while PA intensity [in metabolic equivalent of task (MET)], frequency (times per week), duration (hours per week) and type (weight-bearing exercises or not) were assessed by the Modifiable Activity Questionnaire (MAQ). At baseline, the radiological severity of OA was assessed using Kellgren and Lawrence (KL) grades, comorbidities by the Functional Comorbidity Index (FCI) and perceived vitality with the Short Form Health Survey (SF-36). Group-based multi-trajectory modeling was used to identify subgroups with distinct trajectories of pain, function and PA components. Patients with at least two available measures were included in the analyses. First, trajectory models were estimated with varying number of groups for each of the outcomes separately to clarify the types of distinct trajectories to be represented in the multi-trajectory model. The selection of the optimal models was based on maximization of the Bayesian information criterion, the proportion of patients in each trajectory group (> 5%) and the statistical significance of the equation modeled (intercept only, linear, quadratic or cubic). Then, each outcome trajectories characteristics were included in the multi-trajectory model and the optimal one was chosen according the same selection criteria. Multinomial logistic regressions were performed to identify the predictive baseline characteristics associated with each group and were adjusted for sociodemographic and clinical factors. Results Among the 878 patients, 609 (69.4%) were women, 222 (25.3%) had hip OA, 607 (69.1%) knee OA and 49 (5.6%) both hip and knee OA. Group-based multi-trajectory modeling revealed 3 distinct trajectories of pain, physical function and PA components over 5 years. The first (n = 187, 35.2%) included patients with low pain levels, no functional limitations and who practiced intense PA and weight-bearing exercises. The second (n = 218, 41.0%) included patients with moderate levels of pain and functional limitations and who practiced a less intense PA. Patients belonging to the third group (n = 127, 23.9%) had severe pain, severe functional limitations and a low-intensity practice with rare weight-bearing exercises. Overall, a decrease in intensity, frequency and duration of PA was observed in all groups over 5 years, even in the group of subjects with low pain levels and no functional limitations. In multivariate analyses, female sex (odds ratio [OR] = 2.76, 95% confidence interval [CI] = 1.48–5.17), increasing age (OR = 1.05, 95% CI = 1.01–1.08), a high number of comorbidities (OR = 1.51, 95% CI = 1.21–1.90), a low vitality score (reflecting a high level of fatigue; OR = 0.94, 95% CI = 0.92–0.97) and a high KL grade (reflecting a high radiological severity; OR = 3.42, 95% CI = 1.53–7.65) were associated with the third group membership. Conclusion Based on the 5-year follow-up data, we identified 3 distinct trajectories of pain, physical function and PA components. A high number of comorbidities, a high level of fatigue and a high radiological severity were associated with the trajectories of severe pain and functional limitations. The management of these factors seems important to maintain physical function, limit pain and maintain PA practice in patients with symptomatic knee and/or hip OA.