The aim of the present study was to study effects of feeding in terms of a herbal diet supplement (HD; basal diet plus a supplement with chicory herbs), of early piglet backtest scores and of relative breeding values for skin lesions (RBV-LS) on behavior pattern of weaned pigs during the post-weaning period. In this regard, we implemented a balanced research design, allowing a semi-randomized allocation of pigs in groups with a very similar number of observations for all sub-cells of diet x backtest score x RBV-LS class combinations. With regard to backtest scores, piglets were classified as high-resisting (HR), low-resisting (LR) or intermediate-resisting (IR). RBV-LS were estimated for the sires based on skin lesions from 993 offspring. The sires were categorized into 2 groups with RBV-LS > 100 (favorable genetic value indicating only a few or no lesions) or RBV-LS ≤ 100 (indicating a large number or severe lesions). Video images were analyzed one day after weaning and 5 weeks later, and considered 300 min per monitoring date for the 8 different possible behavior traits resting time (REST), body contact (BCON), initiating fights (IFIGHT), fighting (FIGHT), refusing of fights (RFIGHT), ear or tail biting (BITE), explorative behavior (EXPLORE) and remaining activities (RACT). Finally, 104 pigs with complete observations over the total video monitoring time at both recording dates, and with complete observations for the explanatory variables sire RBV-LS, backtest score and feeding group, were considered for ongoing association studies. The 104 pigs were crosses from matings of Piétrain boars with German Landrace or German Edelschwein sows. The 104 pigs were offspring of 9 different boars (sire lines). Effects of feeding, backtest score and RBV-LS class on video behaviors were inferred via mixed model analyses, implying 8 different runs for the 8 video behavior traits. Least squares means (lsmeans) significantly (P < 0.05) differed between HD and the control feeding group (CON) for FIGHT and BITE, with longer durations of aggressive behavior for CON pigs. Vice versa, lsmeans for REST indicating calm behavior were larger for the HD group. Pigs classified as HR piglets were more aggressive than the LR and IR contemporaries, with significantly higher lsmeans for FIGHT, BITE and EXPLORE. The pigs allocated to the sire RBV-LS > 100 group had significantly (P < 0.05) longer durations for REST and RFIGHT. Vice versa, pigs allocated to the sire RBV-LS ≤ 100 groups spent more with aggressive behavior in terms of BITE and FIGHT. Consequently, boar RBV-LS were favorably correlated with REST, IFIGHT and RACT. In conclusion, the early piglet backtest scores as well as breeding values from a victim perspective (lesion scores) can be used as indicator traits for selection against aggressiveness. Nevertheless, results from the present study are first indications and breeding value correlations are approximations in this regard, which should be validated in ongoing studies based on larger datasets via real genetic correlation estimates between actor and victim traits, and considering social interactions and interactions between all fixed effects simultaneously.