Past research has linked more frequent social contacts with better cognition and slower cognitive decline in older adults. An open question is whether face-to-face and remote contact with one's grandchildren can be beneficial. We analyzed data from the Swedish National Study on Aging and Care in Kungsholmen (SNAC-K) covering a span of 12 years and two age cohorts (young-old < 78, N = 1100; old-old ≥ 78 years, N = 705). We used latent growth curve models, to examine whether frequency of face-to-face or remote grandchild contact was associated with cognitive levels and decline and applied (random intercept) cross-lagged models to investigate if these associations were reciprocal. Face-to-face contact with grandchildren was positively linked to levels of cognition in young-old adults only. We found no associations with cognitive decline. Results of cross-lagged models suggested that grandparents with better cognition had more face-to-face (for young-old adults only) or remote (for old-old adults only) grandchild contact at subsequent waves. However, more grandchild contact was not associated with later cognition. Our findings suggest that grandparents with better cognition engage more with their grandchildren, but that frequency of grandchild contact is not a protective factor against later cognitive decline in older adults.