Humans rapidly locate and recognize human faces, even in complex environments. In the current study, we explored some of the social and perceptual features of faces that may contribute to this ability. We measured infant and adult attention to complex, heterogeneous image arrays containing human and animal faces. Arrays were upright or inverted 180° and in color or grayscale. Infants, aged 3–5 months (n = 51) and 10–11 months (n = 34), viewed 6-item arrays (Experiment 1), whereas adults (n = 120) searched 64-item arrays (Experiment 2). We found that 3- to 5-month-olds already displayed strong own-species biases in face detection—in attention capture, attention holding, and overall detection—suggesting a surprisingly early specialization for human face detection. Furthermore, this remarkable ability was robust, evident even when color and orientation were disrupted, and grew stronger with age. Interestingly, infants’ face detection was reduced by low-level manipulations in a species-specific way, negatively affecting only animal face detection but not affecting human face detection. In contrast, adults’ face detection efficiency was equally reduced by low-level manipulations across species, suggesting potential age differences in own-species face detection. For infants, social relevance (species) may play a more important role than low-level perceptual features, ensuring that infants attend to, connect with, and learn from the people around them. Efficient human face detection during infancy may reflect the uniqueness of own-species faces as a category, perhaps due to their social relevance.