In recent years, a multitude of intergenerational contact programmes and interventions has emerged to counteract ageism among young adults. Research on these programmes and its supposed effect on ageism often start from the assumption that intergenerational contact follows a largely linear process in which a high level of encounters, in the right setting, decreases ageism and negative stereotyping. The purpose of this article is to critically examine this assumption by focussing on the underlying process of intergenerational contact, rather than examining the positive or negative outcome. Using in-depth interviews with 35 young adults, we found that although conditions and mediating factors during the contact do play a role in the outcome of intergroup contact, the process of contact is rather varied and does not follow a linear path. The results reveal that whether or not a positive contact experience translates into a changed group image of older people is related to the positioning of such experience within the young adults' personal frame of reference. We found that this has to do with the young adults having diverse and both positive and negative previous experiences, their grandparent-grandchild relationship, stories from others and personal characteristics. With this study, we point to the complexity of intergenerational contact and highlight potential pathways leading to varying group images of ‘the old’.