The aim of this study was to estimate the genetic association between direct and maternal effects for important economic traits measured early and late in the growth period; daily gain (DG) between birth and 12 weeks of age (DG0-12), between 12 weeks of age and ultrasonic test (at approx. 100 kg; DG12-UT), between birth and ultrasonic test (DG0-UT); and backfat at ultrasonic test (BF). Records from Landrace (L; 39,130), Yorkshire (Y; 33,737), and Hampshire (H; 17,832) pigs born between 1997 and 2001 from the Swedish breeding organization Quality Genetics databank were available. Univariate and bivariate analyses were applied. Two statistical models were used. The first model applied for all traits, included the random effect of herd-birth year-birth month combination, the litter environmental effects, and the direct genetic effects. The second model applied only for DG0-UT and BF, and included also the maternal genetic effects. Based on the first single-trait model, estimates of direct heritability (h2) were 0.12, 0.09, and 0.43 for DG0-12; 0.16, 0.22, and 0.41 for DG12-UT; 0.18, 0.24, and 0.28 for DG0-UT; and 0.37, 0.42, and 0.43 for BF in the L, Y, and H breeds respectively. Estimates of litter environmental effects ranged from 0.05 to 0.17. Based on the second single-trait model, the maternal genetic effects were 0.03, 0.05, and 0.09 for DG0-UT, and 0.02, 0.01, and 0.04 for BF. Estimates of direct h2 from the second model were lower and ranged from 0.14 to 0.42 for all traits across breeds. Moderate-low genetic correlation between DG0-12 and DG12-UT (−0.25 to 0.40) and high correlation between DG12-UT and DG0-UT (0.84 to 0.96) were noted in all three breeds together with less maternal influence on traits expressed late in life. This indicates the possibility to treat early growth rate as a maternal trait, and late or overall growth rate as a direct trait in the breeding evaluation.