In this article, the nonlinear characteristics of magnetostatic forward volume wave (MSFVW)-based guided-wave magnetooptic Bragg cell modulators in bismuth-substituted yttrium iron garnet-gadolinium gallium garnet waveguides using nonuniform bias magnetic field are reported. First, the dispersion characteristics of the MSFVW under nonuniform bias magnetic field are analyzed, and the explicit expression for its bandwidth is determined. The transmission measurements of the MSFVW show that owing to the nonuniform magnetic field, the bandwidth is significantly increased. Next, the results of noncollinear magnetooptic (MO) Bragg diffraction experiments using the MSFVW in the frequency range from 2.0 to 4.0 GHz are presented. Two types of nonlinear process, namely, the four-magnon decay and modulation instabilities, are observed. However, the MO experiments at the carrier frequency of 2.85, 3.10, and 3.25 GHz suggest that the decay instabilities did not play a significant role in the MO interaction because of the larger degree of phase mismatch induced by the satellite waves generated during the nonlinear processes. We find that despite the presence of the decay instabilities, the MO Bragg diffraction characteristics still comply with that predicted by the coupled-mode theory before the nonlinear processes evolve into the modulation instabilities. Once the four-magnon modulation instabilities set in at the threshold powers, the MO Bragg diffraction will incur a drop in diffraction efficiency by as much as 9%. This feature results from perturbation of the satellite waves of smaller wave numbers induced in the modulation instabilities that lead to the MO phase mismatch. A model is established to describe the combined contributions of the initial MSFVW and the excited satellite waves associated with the modulation instabilities to the MO Bragg diffraction characteristics.