Paternal care in mammals can have a wide range of displays. Stallions mostly invest in their offspring indirectly through herd defense and occasional play with the foals, but until recently this relationship had not been analyzed thoroughly. Recent research in semi-feral horses revealed unique social interactions between offspring and their sire, highlighting different attitude of mares and stallions toward the foals and vice versa. Here we present subsequent research focused on the variability of foals' behavior under different social context. Two herds of semi-feral Exmoor ponies in 3 different types of social groups were observed: SMF - stallion + mares + foals (under 1y of age) (1+14+13; 1+14+8, season 2016 and 2017), SMYF - stallion + mares + young (1–2y, both sexes) + foals (1+13+9+10; 1+14+10+13, season 2018) and MYF - mares + young (1–2y females only) + foals (14+6+11; 20+13+18, season 2020). Each type of group was observed for one season, approximately from April to October every 2 weeks (748 h and 8473 interactions in total, all observed years). All herd members were observed directly, by ad libitum sampling method, during sessions lasting 2–4 h throughout the daylight. Differences in the frequencies (χ 2 test) of 4 groups of behavior were calculated (friendly, agonistic, social play and snapping). Probabilities of occurrence in foal's behavior were studied through GLMM considering the effects of sex (colts vs. fillies), type of herd, and type of receiver of the interaction. The results show, that SMYF groups had lower aggressive ( P = 0.003) and social play ( P < 0.001) frequency than the other groups, but higher frequency of friendly and snapping behaviors (both P < 0.001). MYF groups showed significant drop in snapping ( P < 0.001) and increase in social play ( P < 0.001) frequency. For each social system studied, foals consistently preferred their peers for friendly and playful interactions over adults and young. However, fillies preferred colts as play partners over other fillies only when the stallion was present, while in herds with no stallion they significantly switched to same-sex partners. On the contrary, the behavior of foals to mares was constant in all 3 types of groups. The different social context of herds clearly influenced the behavior of the foals. One of the most notable outputs of this study is the influence of stallion presence on fillies’ behavior. The results indicate that the behavior of both sons and daughters may change when they grow up in groups without a father, which is highly relevant considering that these conditions are common in domestic horses’ husbandry.