Parental socio-economic status (SES) is often found to be associated with children's language competence in the first decade of life. To examine the effect of SES on children's vocabulary development, as well as potential compensatory effects of schooling and learning-related activities, we examined the joint and unique effects of parental education, occupational status, and learning environment at home on children's receptive vocabulary competence and growth in early childhood. We used latent growth curve models to assess pre-school receptive vocabulary and growth across primary school. Analyses were based on data from the German National Educational Panel Study (NEPS), a large-scale longitudinal study assessing vocabulary competence and family background from Kindergarten to the 3rd grade of elementary school. To examine the moderating effects of parental education, occupational status, and learning environment at home, we used local structural equation modeling. Results revealed a moderate to strong positive association between parental education and children's receptive vocabulary competence, which fully explained the effect of occupational status on this language skill. With the exception of the activity of reading aloud, we found no effect of learning environment at home. Initially lower performing children showed steeper growth trajectories across school, but rank-orders were relatively stable across time. In summary, the results suggest large initial differences in receptive vocabulary between children from different educational backgrounds, which are reduced, but not fully overcome across elementary school.