This study examined the general public’s perceptions of how the COVID-19 pandemic has impacted the elderly and people with intellectual disabilities as well how these perceptions relate to people’s level of familiarity and contact quality with these groups. A cross-sectional survey was administered to a sample of the Dutch population (n = 1458 and n = 1761, comprising questions related to the elderly and people with intellectual disabilities, respectively). The general public was found to be generally aware of the deleterious impact of the pandemic upon the elderly and people with intellectual disabilities. Specifically, the respondents reported that both groups’ quality of life, physical and mental health, and quality and frequency of social contact was lower than it was prior to COVID-19, in addition to perceiving them as lonelier and less self-reliant. Notably, the impact on the elderly was considered to be greater than that on people with intellectual disabilities. Furthermore, those who had no familiarity with people with intellectual disabilities in real life perceived the impact to be lower than those who had a greater degree of familiarity. These findings have important implications, both for increasing awareness of the pandemic’s negative impact on these vulnerable groups and in terms of sufficiently addressing their specific needs and concerns. The findings also underscore that, particularly during the COVID-19 pandemic, it is important to increase the visibility of groups who already relied more on help and support from others in society prior to the pandemic, such as the elderly and people with intellectual disabilities, via, among other things, self-advocacy, education, and enhanced intergroup contact, in order to be able to sufficiently address their needs during these challenging times.