In composite materials consisting of silver particles and a polypropylene matrix, the silver must migrate to the surface of the materials in order to exert on antibacterial effect. Substrates with various crystal structures may have different interactions with the silver particles, which can significantly impact the antibacterial performance. The research described in this paper investigates the influence of silver particles on the crystallization behavior of materials with varying crystallinity capabilities by selecting various polypropylene-based plastics and elastomers (Homopolymer polypropylene (PPH-T03), random poly(propylene/ethylene) copolymers (PPR-YPR503 and PPR-P340R), polyolefin elastomers (VERSIFY3000 (POE-1) and VERSIFY3401 (POE-2)), and ethylene propylene diene monomer for polymers (EPDM4725 and EPDM4770)). The research results indicated that the silver only entered the amorphous regions of the PP rather than entering the crystal regions, where they can more easily migrate to the material surface, resulting in a higher antibacterial effect. Therefore, the antibacterial degree of PP-Ag was as high as 99.9%. Silver can enter the micro-crystals of both the random copolymer polypropylene, (PPR-YPR503, PPR-1) and the random copolymer polypropylene, (PPR-P340R, PPR-2,) with a stronger effect on PPR-2. More silver entered the micro-crystals of PPR-2, and it was difficult for the silver ions to migrate out of the crystalline regions to exhibit antibacterial properties. The antibacterial degree of PPR-1-Ag was higher than that of PPR-2-Ag (77.8% vs. 66.7%). Similarly, silver had a greater impact on the lamellar structure of the polyolefin elastomer (VERSIFY3401, POE-2) compared to the polyolefin elastomer (VERSIFY3000, POE-1). In comparison with POE-2-Ag, POE-1-Ag exhibited a higher antibacterial degree (79.6% vs. 57.4%). Research on the two types of ethylene propylene diene monomer (EPDM) composite materials indicated that EPDM-1 exhibited a lower Mooney viscosity and poorer flowability. The EPDM-1 tended to encapsulate the silver within its molecular chains, thereby impeding the migration of silver to the surface. The optical performance analysis indicated that silver antibacterial agents had little impact on the optical properties of PP-Ag, PPR-1, PPR-2, POE-1, POE-2 and the EPDMs. We suggest that this research provides theoretical guidance for selecting highly antibacterial polypropylene materials.