Previous research has shown an unintuitive effect of facial expression on perceived age: smiling faces are perceived as older compared to neutral faces of the same people. The aging effect of smiling (AES), which is thought to result from the presence of smile-related wrinkles around the eyes, contradicts the common belief that smiling faces should be perceived as younger, not older. Previous research, however, has focused on faces of young adults, where the absence of inherent, age-related wrinkles and other age signs is offset by the weight of the smile-related wrinkles. In a series of experiments, we tested whether the AES extends to male and female faces in older age groups. We replicated the AES in young adults (20–39) and showed that it disappeared in older adults (60–79) of both genders. For photos of middle-aged adults (40–59), however, AES was found only for male, but not for female faces, who showed fewer and less prominent smile-related wrinkles. The results suggest that a person’s apparent age is perceived in a holistic manner in which age-related cues in the region of the eyes are weighted against age cues in other regions of the face.