The coatings play a crucial role in enhancing the performance of soft magnetic composites (SMCs). This study aims to create coatings for Fe-Ni-Mo SMCs using polysilazane (PSZ) as the precursor. The transformation mechanism of the PSZ/resin mixture is studied by analyzing functional groups and chemical bonds at various temperatures. During the self-polymerization process of PSZ at 350 °C, N–H, C = C–H, Si-H bonds, –CH3, and Si-CH3 groups undergo crosslinking reactions to form C–C and Si-N bonds. When the temperature reaches 600 °C, PSZ polymer cleavage, O-Si-C and Si-N-Si bond dominate in the coating. Additionally, PSZ can effectively enhance the heat resistance of silicone resin through its reaction with Si-OH groups, which has a significant effect on increasing the resistivity of SMCs and reducing high frequency core loss. Optimizing the PSZ content to 0.5 wt% enhances the magnetic properties of Fe-Ni-Mo SMCs, resulting in a μe of 74.0 at 100 kHz and a corresponding core loss (Pcv) of 547.7 mW/cm3 at 100 kHz/100 mT. The findings provide valuable insights for optimizing PSZ content to improve magnetic performance in SMCs.