The nuclear recoil effect on the $^2 P_{3/2}$-state $g$ factor of B-like ions is calculated to first order in the electron-to-nucleus mass ratio $m/M$ in the range $Z=18$--$92$. The calculations are performed by means of the $1/Z$ perturbation theory. Within the independent-electron approximation, the one- and two-electron recoil contributions are evaluated to all orders in the parameter $\alpha Z$ by employing a fully relativistic approach. The interelectronic-interaction correction of first order in $1/Z$ is treated within the Breit approximation. Higher orders in $1/Z$ are partially taken into account by incorporating the screening potential into the zeroth-order Hamiltonian. The most accurate to date theoretical predictions for the nuclear recoil contribution to the bound-electron $g$ factor are obtained.