The nonlinear propagation of a high-amplitude electromagnetic (EM) wave is studied in a magnetized dusty plasma to investigate the three-decay instability in the stimulated scattering of the EM wave due to the density perturbations. The plasma involves the electrons, ions, and negatively charged dust grains, which are immersed in a uniform transverse magnetic field. Using the hydrodynamics plasma model and assuming the plasma ions to be stationary, a set of coupled equations is derived to find a new expression of the plasma frequency in magnetized dusty plasma as well as the instability growth rate of backward scattering. The effects of the external transverse magnetic field on the plasma frequency and the growth rate are reported.