Deformation cracks during continuous casting and hot rolling are easy to occur in the Bi–S based free cutting steel, and the cracks are related to the diffusion of bismuth. The diffusion behavior of bismuth in the steel and the effect of the microstructure and alloying elements were studied by the diffusion annealing experiment combined with SEM, EDS and TEM analysis. The results show that the bismuth tends to diffuse along the grain boundaries (GBs) and the interfaces between the MnS phase and steel matrix, and then enrich to form the bismuth phases. The bismuth phases with lower melting point would be in liquid form at high temperature and resulted in the cracks along the GBs and interfaces. The bismuth diffusion is more likely to occur toward the parallel rolling direction than the vertical rolling direction in the steel because of the pearlite and MnS phases with the long strip shape and banded distribution. The addition of boron and titanium has the effect of inhibiting the diffusion of bismuth in the steel, which is mainly attributed to the faster diffusion rate of boron and the priority to occupy the defect sites of the GBs and the interfaces between the MnS phase and steel matrix.