This study addressed the impact of early and later life environmental enrichment, and their combination, on specific antibody responses and peripheral blood leukocyte subpopulations in pigs. Pigs were kept in either barren (B1) or enriched (E1) housing from birth, and half of the pigs switched to barren or enriched housing on day 47, resulting in four treatment combinations: B1B2, B1E2, E1B2, E1E2). Pigs were immunized with keyhole limpet hemocyanin-conjugated trinitrophenyl (KLH-TNP) on day 74 and 109 to induce primary and secondary antibody responses. Blood samples were taken weekly until day 130, and IgM and IgG antibody responses were measured. Leukocyte subpopulations were measured on day 74 and 130. Time course of the antibody responses was not affected by housing. Early life enrichment increased the IgG response to KLH, particularly the primary one. At day 74 the relative frequency of lymphocytes, DC and SLA-II expression on monocytes were higher in E1 pigs, whereas the percentage of granulocytes tended to be lower in E1 pigs at day 74. Early life enrichment increased the SLA-II expression on monocytes, the granulocyte to lymphocyte ratio, and tended to increase the percentage of granulocytes, but tended to decrease the percentage of monocytes at day 130. Later life enrichment reduced percentages of CD4+CD8α+ T cells before and after immunization and the SLA-II expression on monocytes at day 74, the percentage of granulocytes and the granulocyte to lymphocyte ratio at day 130. Notably, early and later life housing interacted in their effects on several immune parameters. KLH-IgM responses (both primary and secondary) were affected by the interaction between early and later life housing. IgM titers were higher for B1B2 than for E1E2, with the switched animals (B1E2 and E1B2) moving towards the titers of the animals kept in their later life environment from birth onwards. At day 130 the percentage of gamma delta T cells, CD8α+ cytotoxic T cells and DC were not different between pigs kept in B1B2 and E1E2, but there was a clear impact of the switch in housing conditions, particularly for the pigs that changed from barren to enriched housing. We also found effects of coping style (personality) and sex on some immune parameters. In conclusion, both early life and later life enrichment, and, notably a switch in housing conditions influenced specific antibodies and leukocyte subpopulations in pigs. The current study implies that the early life history of animals and the (mis)match with their current environment could thus be of major importance for their immune system. Further research is needed to investigate potential consequences for the pigs’ health.